Attachment 1727165 Details for Bug 1893739 – Finalizer-null-approach-to-delete-the-leftovers

Finalizer-null-approach-to-delete-the-leftovers

after-finalizer-patching-comment-9 (text/plain), 154.88 KB, created by Neha Berry on 2020-11-06 16:35:23 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Neha Berry

Created: 2020-11-06 16:35:23 UTC

Size: 154.88 KB

patch

obsolete

>I1106 16:26:10.413990       1 main.go:66] Version: v4.6.0-202011041933.p0-0-gedd7cec-dirty
>I1106 16:26:10.416379       1 main.go:93] Start NewCSISnapshotController with kubeconfig [] resyncPeriod [1m0s]
>I1106 16:26:10.420116       1 leaderelection.go:243] attempting to acquire leader lease  openshift-cluster-storage-operator/snapshot-controller-leader...
>I1106 16:26:19.046330       1 leaderelection.go:253] successfully acquired lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:26:19.046752       1 reflector.go:207] Starting reflector *v1beta1.VolumeSnapshotClass (1m0s) from github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117
>I1106 16:26:19.047058       1 reflector.go:243] Listing and watching *v1beta1.VolumeSnapshotClass from github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117
>I1106 16:26:19.047085       1 reflector.go:207] Starting reflector *v1.PersistentVolumeClaim (1m0s) from k8s.io/client-go/informers/factory.go:134
>I1106 16:26:19.047097       1 reflector.go:243] Listing and watching *v1.PersistentVolumeClaim from k8s.io/client-go/informers/factory.go:134
>I1106 16:26:19.046940       1 reflector.go:207] Starting reflector *v1beta1.VolumeSnapshot (1m0s) from github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117
>I1106 16:26:19.046975       1 snapshot_controller_base.go:128] Starting snapshot controller
>I1106 16:26:19.047015       1 reflector.go:207] Starting reflector *v1beta1.VolumeSnapshotContent (1m0s) from github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117
>I1106 16:26:19.047420       1 reflector.go:243] Listing and watching *v1beta1.VolumeSnapshotContent from github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117
>I1106 16:26:19.047284       1 reflector.go:243] Listing and watching *v1beta1.VolumeSnapshot from github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117
>I1106 16:26:19.056711       1 snapshot_controller_base.go:175] enqueued "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" for sync
>I1106 16:26:19.056820       1 snapshot_controller_base.go:175] enqueued "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" for sync
>I1106 16:26:19.057628       1 snapshot_controller_base.go:158] enqueued "default/test-cephfs-snapshot" for sync
>I1106 16:26:19.057644       1 snapshot_controller_base.go:158] enqueued "default/test-rbd-snapshot" for sync
>I1106 16:26:19.147578       1 shared_informer.go:270] caches populated
>I1106 16:26:19.147725       1 util.go:230] storeObjectUpdate: adding snapshot "default/test-cephfs-snapshot", version 1565552
>I1106 16:26:19.147757       1 util.go:230] storeObjectUpdate: adding snapshot "default/test-rbd-snapshot", version 1565563
>I1106 16:26:19.147784       1 util.go:230] storeObjectUpdate: adding content "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", version 1565551
>I1106 16:26:19.147811       1 util.go:230] storeObjectUpdate: adding content "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", version 1565561
>I1106 16:26:19.147855       1 snapshot_controller_base.go:471] controller initialized
>I1106 16:26:19.147951       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-cephfs-snapshot]
>I1106 16:26:19.147971       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:26:19.147993       1 util.go:258] storeObjectUpdate updating content "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" with version 1565551
>I1106 16:26:19.148006       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: content is bound to snapshot default/test-cephfs-snapshot
>I1106 16:26:19.148009       1 snapshot_controller.go:88] syncContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: check if we should add invalid label on content
>I1106 16:26:19.148016       1 snapshot_controller.go:1368] getSnapshotFromStore: snapshot default/test-cephfs-snapshot found
>I1106 16:26:19.148030       1 snapshot_controller.go:989] needsUpdateSnapshotStatus[default/test-cephfs-snapshot]
>I1106 16:26:19.148039       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [default/test-cephfs-snapshot]: update snapshot status to true if needed.
>I1106 16:26:19.147952       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:26:19.148065       1 util.go:258] storeObjectUpdate updating content "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" with version 1565561
>I1106 16:26:19.148078       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: content is bound to snapshot default/test-rbd-snapshot
>I1106 16:26:19.148085       1 snapshot_controller.go:88] syncContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: check if we should add invalid label on content
>I1106 16:26:19.148090       1 snapshot_controller.go:1368] getSnapshotFromStore: snapshot default/test-rbd-snapshot found
>I1106 16:26:19.148095       1 snapshot_controller.go:989] needsUpdateSnapshotStatus[default/test-rbd-snapshot]
>I1106 16:26:19.148109       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [default/test-rbd-snapshot]: update snapshot status to true if needed.
>I1106 16:26:19.147969       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-rbd-snapshot]
>I1106 16:26:19.148142       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-rbd-snapshot]
>I1106 16:26:19.148158       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-rbd-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>I1106 16:26:19.148183       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>I1106 16:26:19.148067       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-cephfs-snapshot]
>I1106 16:26:19.148251       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-cephfs-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>I1106 16:26:19.148259       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>E1106 16:26:19.148207       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-rbdplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-rbdplugin-snapclass\" not found"
>E1106 16:26:19.148306       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-rbdplugin-snapclass" not found
>I1106 16:26:19.148323       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-rbd-snapshot]
>I1106 16:26:19.148340       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-rbd-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000128690} is already set
>I1106 16:26:19.148380       1 snapshot_controller_base.go:220] Snapshot "default/test-rbd-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:26:19.148394       1 snapshot_controller_base.go:222] Updating snapshot "default/test-rbd-snapshot"
>E1106 16:26:19.148270       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-cephfsplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-cephfsplugin-snapclass\" not found"
>E1106 16:26:19.148433       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-cephfsplugin-snapclass" not found
>I1106 16:26:19.148441       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-cephfs-snapshot]
>I1106 16:26:19.148445       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-cephfs-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc0001285d0} is already set
>I1106 16:26:19.148464       1 snapshot_controller_base.go:220] Snapshot "default/test-cephfs-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:26:19.148471       1 snapshot_controller_base.go:222] Updating snapshot "default/test-cephfs-snapshot"
>I1106 16:26:19.148476       1 snapshot_controller_base.go:358] updateSnapshot "default/test-cephfs-snapshot"
>I1106 16:26:19.148483       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:26:19.148494       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:26:19.148498       1 snapshot_controller.go:182] syncSnapshot [default/test-cephfs-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:26:19.148514       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:19.148522       1 snapshot_controller.go:191] syncSnapshot[default/test-cephfs-snapshot]: check if we should add invalid label on snapshot
>I1106 16:26:19.148528       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:26:19.148533       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-cephfs-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:26:19.148538       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:26:19.148548       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-cephfs-snapshot
>I1106 16:26:19.148554       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-cephfs-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:26:19.148559       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418].
>I1106 16:26:19.148407       1 snapshot_controller_base.go:358] updateSnapshot "default/test-rbd-snapshot"
>I1106 16:26:19.148588       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:26:19.148645       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:26:19.148659       1 snapshot_controller.go:182] syncSnapshot [default/test-rbd-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:26:19.148678       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:19.148692       1 snapshot_controller.go:191] syncSnapshot[default/test-rbd-snapshot]: check if we should add invalid label on snapshot
>I1106 16:26:19.148712       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:26:19.148726       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-rbd-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:26:19.148739       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:26:19.148755       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-rbd-snapshot
>I1106 16:26:19.148768       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-rbd-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:26:19.148797       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b].
>I1106 16:26:19.246912       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-rbd-snapshot]
>I1106 16:26:19.246942       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:19.268657       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:26:19.268685       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-rbd-snapshot
>I1106 16:26:19.268724       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-rbd-snapshot]
>I1106 16:26:19.268729       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-rbd-snapshot]
>I1106 16:26:19.268737       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-rbd-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>I1106 16:26:19.268742       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>E1106 16:26:19.268748       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-rbdplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-rbdplugin-snapclass\" not found"
>E1106 16:26:19.268758       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-rbdplugin-snapclass" not found
>I1106 16:26:19.268764       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-rbd-snapshot]
>I1106 16:26:19.268767       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-rbd-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000128690} is already set
>I1106 16:26:19.268790       1 snapshot_controller_base.go:220] Snapshot "default/test-rbd-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:26:19.268795       1 snapshot_controller_base.go:222] Updating snapshot "default/test-rbd-snapshot"
>I1106 16:26:19.268799       1 snapshot_controller_base.go:358] updateSnapshot "default/test-rbd-snapshot"
>I1106 16:26:19.268803       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:26:19.268811       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:26:19.268818       1 snapshot_controller.go:182] syncSnapshot [default/test-rbd-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:26:19.268827       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:19.268832       1 snapshot_controller.go:191] syncSnapshot[default/test-rbd-snapshot]: check if we should add invalid label on snapshot
>I1106 16:26:19.268838       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:26:19.268843       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-rbd-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:26:19.268849       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:26:19.268860       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-rbd-snapshot
>I1106 16:26:19.268866       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-rbd-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:26:19.268870       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b].
>I1106 16:26:19.439117       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:26:19.456615       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-rbd-snapshot]
>I1106 16:26:19.456754       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:19.484706       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:26:19.484739       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-rbd-snapshot
>I1106 16:26:19.492342       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-cephfs-snapshot]
>I1106 16:26:19.492376       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:19.560162       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:26:19.560284       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-cephfs-snapshot
>I1106 16:26:19.560359       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-cephfs-snapshot]
>I1106 16:26:19.560381       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-cephfs-snapshot]
>I1106 16:26:19.560400       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-cephfs-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>I1106 16:26:19.560428       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>E1106 16:26:19.560446       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-cephfsplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-cephfsplugin-snapclass\" not found"
>E1106 16:26:19.560463       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-cephfsplugin-snapclass" not found
>I1106 16:26:19.560481       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-cephfs-snapshot]
>I1106 16:26:19.560578       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-cephfs-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc0001285d0} is already set
>I1106 16:26:19.560616       1 snapshot_controller_base.go:220] Snapshot "default/test-cephfs-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:26:19.560659       1 snapshot_controller_base.go:222] Updating snapshot "default/test-cephfs-snapshot"
>I1106 16:26:19.560675       1 snapshot_controller_base.go:358] updateSnapshot "default/test-cephfs-snapshot"
>I1106 16:26:19.560690       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:26:19.560708       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:26:19.560725       1 snapshot_controller.go:182] syncSnapshot [default/test-cephfs-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:26:19.560765       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:19.560784       1 snapshot_controller.go:191] syncSnapshot[default/test-cephfs-snapshot]: check if we should add invalid label on snapshot
>I1106 16:26:19.560800       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:26:19.560824       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-cephfs-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:26:19.560838       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:26:19.560863       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-cephfs-snapshot
>I1106 16:26:19.560877       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-cephfs-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:26:19.560890       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418].
>I1106 16:26:19.589992       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-cephfs-snapshot]
>I1106 16:26:19.590030       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:19.621300       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:26:19.621427       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-cephfs-snapshot
>I1106 16:26:24.462238       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:26:29.596321       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:26:38.848707       1 reflector.go:515] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PersistentVolumeClaim total 0 items received
>I1106 16:26:41.081373       1 reflector.go:422] k8s.io/client-go/informers/factory.go:134: watch of *v1.PersistentVolumeClaim closed with: too old resource version: 1624938 (1625346)
>I1106 16:26:41.405091       1 reflector.go:515] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: Watch close - *v1beta1.VolumeSnapshotContent total 0 items received
>I1106 16:26:41.412854       1 reflector.go:515] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: Watch close - *v1beta1.VolumeSnapshotClass total 0 items received
>I1106 16:26:41.413059       1 reflector.go:515] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: Watch close - *v1beta1.VolumeSnapshot total 0 items received
>I1106 16:26:41.564147       1 reflector.go:422] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: watch of *v1beta1.VolumeSnapshotContent closed with: too old resource version: 1624615 (1625346)
>I1106 16:26:41.904251       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:26:41.934322       1 reflector.go:243] Listing and watching *v1.PersistentVolumeClaim from k8s.io/client-go/informers/factory.go:134
>I1106 16:26:41.981906       1 reflector.go:422] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: watch of *v1beta1.VolumeSnapshot closed with: too old resource version: 1624605 (1625354)
>I1106 16:26:42.777938       1 reflector.go:243] Listing and watching *v1beta1.VolumeSnapshotContent from github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117
>I1106 16:26:42.808493       1 snapshot_controller_base.go:175] enqueued "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" for sync
>I1106 16:26:42.808537       1 snapshot_controller_base.go:175] enqueued "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" for sync
>I1106 16:26:42.808549       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:26:42.808561       1 util.go:258] storeObjectUpdate updating content "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" with version 1565551
>I1106 16:26:42.808572       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: content is bound to snapshot default/test-cephfs-snapshot
>I1106 16:26:42.808577       1 snapshot_controller.go:88] syncContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: check if we should add invalid label on content
>I1106 16:26:42.808585       1 snapshot_controller.go:1368] getSnapshotFromStore: snapshot default/test-cephfs-snapshot found
>I1106 16:26:42.808606       1 snapshot_controller.go:989] needsUpdateSnapshotStatus[default/test-cephfs-snapshot]
>I1106 16:26:42.808613       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [default/test-cephfs-snapshot]: update snapshot status to true if needed.
>I1106 16:26:42.808633       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:26:42.808639       1 util.go:258] storeObjectUpdate updating content "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" with version 1565561
>I1106 16:26:42.808646       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: content is bound to snapshot default/test-rbd-snapshot
>I1106 16:26:42.808650       1 snapshot_controller.go:88] syncContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: check if we should add invalid label on content
>I1106 16:26:42.808654       1 snapshot_controller.go:1368] getSnapshotFromStore: snapshot default/test-rbd-snapshot found
>I1106 16:26:42.808659       1 snapshot_controller.go:989] needsUpdateSnapshotStatus[default/test-rbd-snapshot]
>I1106 16:26:42.808663       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [default/test-rbd-snapshot]: update snapshot status to true if needed.
>I1106 16:26:42.808688       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-cephfs-snapshot]
>I1106 16:26:42.808693       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-cephfs-snapshot]
>I1106 16:26:42.808700       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-cephfs-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>I1106 16:26:42.808709       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>E1106 16:26:42.808718       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-cephfsplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-cephfsplugin-snapclass\" not found"
>E1106 16:26:42.808763       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-cephfsplugin-snapclass" not found
>I1106 16:26:42.808772       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-cephfs-snapshot]
>I1106 16:26:42.808775       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-cephfs-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc0001285d0} is already set
>I1106 16:26:42.808800       1 snapshot_controller_base.go:220] Snapshot "default/test-cephfs-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:26:42.808809       1 snapshot_controller_base.go:222] Updating snapshot "default/test-cephfs-snapshot"
>I1106 16:26:42.808814       1 snapshot_controller_base.go:358] updateSnapshot "default/test-cephfs-snapshot"
>I1106 16:26:42.808829       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:26:42.808847       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:26:42.808852       1 snapshot_controller.go:182] syncSnapshot [default/test-cephfs-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:26:42.808864       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:42.808876       1 snapshot_controller.go:191] syncSnapshot[default/test-cephfs-snapshot]: check if we should add invalid label on snapshot
>I1106 16:26:42.808888       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:26:42.808893       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-cephfs-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:26:42.808912       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:26:42.808928       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-cephfs-snapshot
>I1106 16:26:42.808934       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-cephfs-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:26:42.808940       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418].
>I1106 16:26:42.809268       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-rbd-snapshot]
>I1106 16:26:42.809281       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-rbd-snapshot]
>I1106 16:26:42.809289       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-rbd-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>I1106 16:26:42.809293       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>E1106 16:26:42.809300       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-rbdplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-rbdplugin-snapclass\" not found"
>E1106 16:26:42.809332       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-rbdplugin-snapclass" not found
>I1106 16:26:42.809339       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-rbd-snapshot]
>I1106 16:26:42.809343       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-rbd-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000128690} is already set
>I1106 16:26:42.809366       1 snapshot_controller_base.go:220] Snapshot "default/test-rbd-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:26:42.809373       1 snapshot_controller_base.go:222] Updating snapshot "default/test-rbd-snapshot"
>I1106 16:26:42.809378       1 snapshot_controller_base.go:358] updateSnapshot "default/test-rbd-snapshot"
>I1106 16:26:42.809388       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:26:42.809411       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:26:42.809420       1 snapshot_controller.go:182] syncSnapshot [default/test-rbd-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:26:42.809430       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:42.809438       1 snapshot_controller.go:191] syncSnapshot[default/test-rbd-snapshot]: check if we should add invalid label on snapshot
>I1106 16:26:42.809443       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:26:42.809447       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-rbd-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:26:42.809453       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:26:42.809460       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-rbd-snapshot
>I1106 16:26:42.809467       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-rbd-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:26:42.809470       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b].
>I1106 16:26:43.432993       1 reflector.go:243] Listing and watching *v1beta1.VolumeSnapshot from github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117
>I1106 16:26:43.436111       1 snapshot_controller_base.go:158] enqueued "default/test-cephfs-snapshot" for sync
>I1106 16:26:43.436154       1 snapshot_controller_base.go:158] enqueued "default/test-rbd-snapshot" for sync
>I1106 16:26:44.688658       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-rbd-snapshot]
>I1106 16:26:44.688693       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:44.737067       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:26:44.737089       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-rbd-snapshot
>I1106 16:26:44.737115       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-rbd-snapshot]
>I1106 16:26:44.737120       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-rbd-snapshot]
>I1106 16:26:44.737128       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-rbd-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>I1106 16:26:44.737134       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>E1106 16:26:44.737141       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-rbdplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-rbdplugin-snapclass\" not found"
>E1106 16:26:44.737153       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-rbdplugin-snapclass" not found
>I1106 16:26:44.737162       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-rbd-snapshot]
>I1106 16:26:44.737166       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-rbd-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483dd0} is already set
>I1106 16:26:44.737190       1 snapshot_controller_base.go:220] Snapshot "default/test-rbd-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:26:44.737195       1 snapshot_controller_base.go:222] Updating snapshot "default/test-rbd-snapshot"
>I1106 16:26:44.737199       1 snapshot_controller_base.go:358] updateSnapshot "default/test-rbd-snapshot"
>I1106 16:26:44.737204       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:26:44.737214       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:26:44.737218       1 snapshot_controller.go:182] syncSnapshot [default/test-rbd-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:26:44.737227       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:44.737232       1 snapshot_controller.go:191] syncSnapshot[default/test-rbd-snapshot]: check if we should add invalid label on snapshot
>I1106 16:26:44.737238       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:26:44.737244       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-rbd-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:26:44.737251       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:26:44.737263       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-rbd-snapshot
>I1106 16:26:44.737268       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-rbd-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:26:44.737272       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b].
>I1106 16:26:46.921858       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:26:47.275371       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-rbd-snapshot]
>I1106 16:26:47.275400       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:47.353471       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:26:47.353509       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-rbd-snapshot
>I1106 16:26:47.353676       1 reflector.go:422] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: watch of *v1beta1.VolumeSnapshotClass closed with: too old resource version: 1624641 (1625396)
>I1106 16:26:47.494284       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-cephfs-snapshot]
>I1106 16:26:47.494969       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:47.839653       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:26:47.839679       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-cephfs-snapshot
>I1106 16:26:47.839712       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-cephfs-snapshot]
>I1106 16:26:47.839717       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-cephfs-snapshot]
>I1106 16:26:47.839726       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-cephfs-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>I1106 16:26:47.839731       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>E1106 16:26:47.839737       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-cephfsplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-cephfsplugin-snapclass\" not found"
>E1106 16:26:47.839772       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-cephfsplugin-snapclass" not found
>I1106 16:26:47.839781       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-cephfs-snapshot]
>I1106 16:26:47.839785       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-cephfs-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483d10} is already set
>I1106 16:26:47.839814       1 snapshot_controller_base.go:220] Snapshot "default/test-cephfs-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:26:47.839819       1 snapshot_controller_base.go:222] Updating snapshot "default/test-cephfs-snapshot"
>I1106 16:26:47.839823       1 snapshot_controller_base.go:358] updateSnapshot "default/test-cephfs-snapshot"
>I1106 16:26:47.839827       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:26:47.839835       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:26:47.839839       1 snapshot_controller.go:182] syncSnapshot [default/test-cephfs-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:26:47.839848       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:47.839853       1 snapshot_controller.go:191] syncSnapshot[default/test-cephfs-snapshot]: check if we should add invalid label on snapshot
>I1106 16:26:47.839860       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:26:47.840072       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-cephfs-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:26:47.840084       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:26:47.840096       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-cephfs-snapshot
>I1106 16:26:47.840105       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-cephfs-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:26:47.840112       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418].
>I1106 16:26:48.064223       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-cephfs-snapshot]
>I1106 16:26:48.064255       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:26:48.161808       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:26:48.161835       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-cephfs-snapshot
>I1106 16:26:48.529090       1 reflector.go:243] Listing and watching *v1beta1.VolumeSnapshotClass from github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117
>I1106 16:26:51.932942       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:27:01.997543       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:27:07.009742       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:27:12.028840       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:27:17.091016       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:27:27.020368       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:27:32.043399       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:27:37.095730       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:27:42.150689       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:27:42.808583       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:27:42.808720       1 snapshot_controller_base.go:175] enqueued "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" for sync
>I1106 16:27:42.808750       1 snapshot_controller_base.go:175] enqueued "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" for sync
>I1106 16:27:42.808789       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:27:42.808798       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:27:42.808811       1 util.go:258] storeObjectUpdate updating content "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" with version 1565561
>I1106 16:27:42.808852       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: content is bound to snapshot default/test-rbd-snapshot
>I1106 16:27:42.808827       1 util.go:258] storeObjectUpdate updating content "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" with version 1565551
>I1106 16:27:42.808865       1 snapshot_controller.go:88] syncContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: check if we should add invalid label on content
>I1106 16:27:42.808896       1 snapshot_controller.go:1368] getSnapshotFromStore: snapshot default/test-rbd-snapshot found
>I1106 16:27:42.808913       1 snapshot_controller.go:989] needsUpdateSnapshotStatus[default/test-rbd-snapshot]
>I1106 16:27:42.808926       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [default/test-rbd-snapshot]: update snapshot status to true if needed.
>I1106 16:27:42.808874       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: content is bound to snapshot default/test-cephfs-snapshot
>I1106 16:27:42.808950       1 snapshot_controller.go:88] syncContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: check if we should add invalid label on content
>I1106 16:27:42.808955       1 snapshot_controller.go:1368] getSnapshotFromStore: snapshot default/test-cephfs-snapshot found
>I1106 16:27:42.808959       1 snapshot_controller.go:989] needsUpdateSnapshotStatus[default/test-cephfs-snapshot]
>I1106 16:27:42.808964       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [default/test-cephfs-snapshot]: update snapshot status to true if needed.
>I1106 16:27:42.808982       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-rbd-snapshot]
>I1106 16:27:42.808986       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-rbd-snapshot]
>I1106 16:27:42.808993       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-rbd-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>I1106 16:27:42.809024       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>E1106 16:27:42.809033       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-rbdplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-rbdplugin-snapclass\" not found"
>I1106 16:27:42.809007       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-cephfs-snapshot]
>I1106 16:27:42.809086       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-cephfs-snapshot]
>I1106 16:27:42.809090       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-cephfs-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>I1106 16:27:42.809092       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>E1106 16:27:42.809097       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-cephfsplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-cephfsplugin-snapclass\" not found"
>E1106 16:27:42.809113       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-cephfsplugin-snapclass" not found
>I1106 16:27:42.809123       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-cephfs-snapshot]
>I1106 16:27:42.809128       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-cephfs-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483d10} is already set
>I1106 16:27:42.809194       1 snapshot_controller_base.go:220] Snapshot "default/test-cephfs-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:27:42.809202       1 snapshot_controller_base.go:222] Updating snapshot "default/test-cephfs-snapshot"
>I1106 16:27:42.809207       1 snapshot_controller_base.go:358] updateSnapshot "default/test-cephfs-snapshot"
>I1106 16:27:42.809214       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:27:42.809226       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:27:42.809229       1 snapshot_controller.go:182] syncSnapshot [default/test-cephfs-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:27:42.809238       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:27:42.809248       1 snapshot_controller.go:191] syncSnapshot[default/test-cephfs-snapshot]: check if we should add invalid label on snapshot
>I1106 16:27:42.809255       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:27:42.809259       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-cephfs-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:27:42.809264       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:27:42.809274       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-cephfs-snapshot
>I1106 16:27:42.809280       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-cephfs-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:27:42.809285       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418].
>E1106 16:27:42.809070       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-rbdplugin-snapclass" not found
>I1106 16:27:42.809664       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-rbd-snapshot]
>I1106 16:27:42.809670       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-rbd-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483dd0} is already set
>I1106 16:27:42.809686       1 snapshot_controller_base.go:220] Snapshot "default/test-rbd-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:27:42.809694       1 snapshot_controller_base.go:222] Updating snapshot "default/test-rbd-snapshot"
>I1106 16:27:42.809697       1 snapshot_controller_base.go:358] updateSnapshot "default/test-rbd-snapshot"
>I1106 16:27:42.809704       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:27:42.809728       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:27:42.809732       1 snapshot_controller.go:182] syncSnapshot [default/test-rbd-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:27:42.809769       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:27:42.809775       1 snapshot_controller.go:191] syncSnapshot[default/test-rbd-snapshot]: check if we should add invalid label on snapshot
>I1106 16:27:42.809781       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:27:42.809793       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-rbd-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:27:42.809801       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:27:42.809808       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-rbd-snapshot
>I1106 16:27:42.809818       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-rbd-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:27:42.809824       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b].
>I1106 16:27:42.816181       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-cephfs-snapshot]
>I1106 16:27:42.816208       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:27:42.822689       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:27:42.822769       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-cephfs-snapshot
>I1106 16:27:42.836837       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-rbd-snapshot]
>I1106 16:27:42.836864       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:27:42.844009       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:27:42.844040       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-rbd-snapshot
>I1106 16:27:43.436264       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:27:43.436345       1 snapshot_controller_base.go:158] enqueued "default/test-cephfs-snapshot" for sync
>I1106 16:27:43.436361       1 snapshot_controller_base.go:158] enqueued "default/test-rbd-snapshot" for sync
>I1106 16:27:43.436403       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-cephfs-snapshot]
>I1106 16:27:43.436413       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-cephfs-snapshot]
>I1106 16:27:43.436420       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-cephfs-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>I1106 16:27:43.436427       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>I1106 16:27:43.436442       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-rbd-snapshot]
>I1106 16:27:43.436472       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-rbd-snapshot]
>I1106 16:27:43.436489       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-rbd-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>I1106 16:27:43.436522       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>E1106 16:27:43.436544       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-rbdplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-rbdplugin-snapclass\" not found"
>E1106 16:27:43.436575       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-rbdplugin-snapclass" not found
>I1106 16:27:43.436595       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-rbd-snapshot]
>I1106 16:27:43.436608       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-rbd-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483dd0} is already set
>I1106 16:27:43.436638       1 snapshot_controller_base.go:220] Snapshot "default/test-rbd-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:27:43.436652       1 snapshot_controller_base.go:222] Updating snapshot "default/test-rbd-snapshot"
>I1106 16:27:43.436667       1 snapshot_controller_base.go:358] updateSnapshot "default/test-rbd-snapshot"
>I1106 16:27:43.436699       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:27:43.436718       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:27:43.436731       1 snapshot_controller.go:182] syncSnapshot [default/test-rbd-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:27:43.436748       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>E1106 16:27:43.436435       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-cephfsplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-cephfsplugin-snapclass\" not found"
>I1106 16:27:43.436767       1 snapshot_controller.go:191] syncSnapshot[default/test-rbd-snapshot]: check if we should add invalid label on snapshot
>I1106 16:27:43.436797       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:27:43.436917       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-rbd-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:27:43.436933       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>E1106 16:27:43.436779       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-cephfsplugin-snapclass" not found
>I1106 16:27:43.436960       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-cephfs-snapshot]
>I1106 16:27:43.436966       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-cephfs-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483d10} is already set
>I1106 16:27:43.436975       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-rbd-snapshot
>I1106 16:27:43.437000       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-rbd-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:27:43.437068       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b].
>I1106 16:27:43.436987       1 snapshot_controller_base.go:220] Snapshot "default/test-cephfs-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:27:43.437114       1 snapshot_controller_base.go:222] Updating snapshot "default/test-cephfs-snapshot"
>I1106 16:27:43.437119       1 snapshot_controller_base.go:358] updateSnapshot "default/test-cephfs-snapshot"
>I1106 16:27:43.437128       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:27:43.437138       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:27:43.437141       1 snapshot_controller.go:182] syncSnapshot [default/test-cephfs-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:27:43.437150       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:27:43.437159       1 snapshot_controller.go:191] syncSnapshot[default/test-cephfs-snapshot]: check if we should add invalid label on snapshot
>I1106 16:27:43.437164       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:27:43.437169       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-cephfs-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:27:43.437183       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:27:43.437191       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-cephfs-snapshot
>I1106 16:27:43.437198       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-cephfs-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:27:43.437202       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418].
>I1106 16:27:43.446663       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-rbd-snapshot]
>I1106 16:27:43.446697       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:27:43.448579       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-cephfs-snapshot]
>I1106 16:27:43.448601       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:27:43.451786       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:27:43.451811       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-rbd-snapshot
>I1106 16:27:43.452599       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:27:43.452612       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-cephfs-snapshot
>I1106 16:27:47.190098       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:27:52.200441       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:27:57.217098       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:28:02.228904       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:28:07.239130       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:28:12.250022       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:28:17.260101       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:28:22.268598       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:28:27.287356       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:28:32.298221       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:28:37.310825       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:28:42.325091       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:28:42.808816       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:28:42.808897       1 snapshot_controller_base.go:175] enqueued "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" for sync
>I1106 16:28:42.808913       1 snapshot_controller_base.go:175] enqueued "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" for sync
>I1106 16:28:42.808925       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:28:42.808936       1 util.go:258] storeObjectUpdate updating content "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" with version 1565561
>I1106 16:28:42.808945       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: content is bound to snapshot default/test-rbd-snapshot
>I1106 16:28:42.808949       1 snapshot_controller.go:88] syncContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: check if we should add invalid label on content
>I1106 16:28:42.808955       1 snapshot_controller.go:1368] getSnapshotFromStore: snapshot default/test-rbd-snapshot found
>I1106 16:28:42.808961       1 snapshot_controller.go:989] needsUpdateSnapshotStatus[default/test-rbd-snapshot]
>I1106 16:28:42.808965       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [default/test-rbd-snapshot]: update snapshot status to true if needed.
>I1106 16:28:42.808980       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:28:42.808987       1 util.go:258] storeObjectUpdate updating content "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" with version 1565551
>I1106 16:28:42.808992       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: content is bound to snapshot default/test-cephfs-snapshot
>I1106 16:28:42.808995       1 snapshot_controller.go:88] syncContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: check if we should add invalid label on content
>I1106 16:28:42.808997       1 snapshot_controller.go:1368] getSnapshotFromStore: snapshot default/test-cephfs-snapshot found
>I1106 16:28:42.809001       1 snapshot_controller.go:989] needsUpdateSnapshotStatus[default/test-cephfs-snapshot]
>I1106 16:28:42.809004       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [default/test-cephfs-snapshot]: update snapshot status to true if needed.
>I1106 16:28:42.809020       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-rbd-snapshot]
>I1106 16:28:42.809039       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-rbd-snapshot]
>I1106 16:28:42.809093       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-rbd-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>I1106 16:28:42.809103       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>E1106 16:28:42.809122       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-rbdplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-rbdplugin-snapclass\" not found"
>E1106 16:28:42.809165       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-rbdplugin-snapclass" not found
>I1106 16:28:42.809175       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-rbd-snapshot]
>I1106 16:28:42.809178       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-rbd-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483dd0} is already set
>I1106 16:28:42.809206       1 snapshot_controller_base.go:220] Snapshot "default/test-rbd-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:28:42.809211       1 snapshot_controller_base.go:222] Updating snapshot "default/test-rbd-snapshot"
>I1106 16:28:42.809215       1 snapshot_controller_base.go:358] updateSnapshot "default/test-rbd-snapshot"
>I1106 16:28:42.809222       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:28:42.809230       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:28:42.809234       1 snapshot_controller.go:182] syncSnapshot [default/test-rbd-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:28:42.809242       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:28:42.809250       1 snapshot_controller.go:191] syncSnapshot[default/test-rbd-snapshot]: check if we should add invalid label on snapshot
>I1106 16:28:42.809257       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:28:42.809261       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-rbd-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:28:42.809266       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:28:42.809275       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-rbd-snapshot
>I1106 16:28:42.809280       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-rbd-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:28:42.809283       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b].
>I1106 16:28:42.809693       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-cephfs-snapshot]
>I1106 16:28:42.809706       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-cephfs-snapshot]
>I1106 16:28:42.809713       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-cephfs-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>I1106 16:28:42.809717       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>E1106 16:28:42.809752       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-cephfsplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-cephfsplugin-snapclass\" not found"
>E1106 16:28:42.809761       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-cephfsplugin-snapclass" not found
>I1106 16:28:42.809772       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-cephfs-snapshot]
>I1106 16:28:42.809776       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-cephfs-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483d10} is already set
>I1106 16:28:42.809798       1 snapshot_controller_base.go:220] Snapshot "default/test-cephfs-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:28:42.809803       1 snapshot_controller_base.go:222] Updating snapshot "default/test-cephfs-snapshot"
>I1106 16:28:42.809806       1 snapshot_controller_base.go:358] updateSnapshot "default/test-cephfs-snapshot"
>I1106 16:28:42.809813       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:28:42.809820       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:28:42.809824       1 snapshot_controller.go:182] syncSnapshot [default/test-cephfs-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:28:42.809836       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:28:42.809842       1 snapshot_controller.go:191] syncSnapshot[default/test-cephfs-snapshot]: check if we should add invalid label on snapshot
>I1106 16:28:42.809847       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:28:42.809869       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-cephfs-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:28:42.809878       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:28:42.809886       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-cephfs-snapshot
>I1106 16:28:42.809894       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-cephfs-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:28:42.809899       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418].
>I1106 16:28:42.817417       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-cephfs-snapshot]
>I1106 16:28:42.817445       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:28:42.817721       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-rbd-snapshot]
>I1106 16:28:42.817735       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:28:42.825376       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:28:42.825397       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-rbd-snapshot
>I1106 16:28:42.825609       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:28:42.825621       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-cephfs-snapshot
>I1106 16:28:43.436386       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:28:43.436475       1 snapshot_controller_base.go:158] enqueued "default/test-cephfs-snapshot" for sync
>I1106 16:28:43.436488       1 snapshot_controller_base.go:158] enqueued "default/test-rbd-snapshot" for sync
>I1106 16:28:43.436516       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-cephfs-snapshot]
>I1106 16:28:43.436522       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-cephfs-snapshot]
>I1106 16:28:43.436528       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-cephfs-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>I1106 16:28:43.436535       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>E1106 16:28:43.436543       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-cephfsplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-cephfsplugin-snapclass\" not found"
>E1106 16:28:43.436556       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-cephfsplugin-snapclass" not found
>I1106 16:28:43.436565       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-cephfs-snapshot]
>I1106 16:28:43.436569       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-cephfs-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483d10} is already set
>I1106 16:28:43.436608       1 snapshot_controller_base.go:220] Snapshot "default/test-cephfs-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:28:43.436617       1 snapshot_controller_base.go:222] Updating snapshot "default/test-cephfs-snapshot"
>I1106 16:28:43.436622       1 snapshot_controller_base.go:358] updateSnapshot "default/test-cephfs-snapshot"
>I1106 16:28:43.436630       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:28:43.436640       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:28:43.436644       1 snapshot_controller.go:182] syncSnapshot [default/test-cephfs-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:28:43.436654       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:28:43.436660       1 snapshot_controller.go:191] syncSnapshot[default/test-cephfs-snapshot]: check if we should add invalid label on snapshot
>I1106 16:28:43.436775       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:28:43.436783       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-cephfs-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:28:43.436791       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:28:43.436804       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-cephfs-snapshot
>I1106 16:28:43.436811       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-cephfs-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:28:43.436815       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418].
>I1106 16:28:43.437286       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-rbd-snapshot]
>I1106 16:28:43.437298       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-rbd-snapshot]
>I1106 16:28:43.437306       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-rbd-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>I1106 16:28:43.437310       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>E1106 16:28:43.437316       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-rbdplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-rbdplugin-snapclass\" not found"
>E1106 16:28:43.437378       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-rbdplugin-snapclass" not found
>I1106 16:28:43.437387       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-rbd-snapshot]
>I1106 16:28:43.437390       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-rbd-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483dd0} is already set
>I1106 16:28:43.437407       1 snapshot_controller_base.go:220] Snapshot "default/test-rbd-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:28:43.437412       1 snapshot_controller_base.go:222] Updating snapshot "default/test-rbd-snapshot"
>I1106 16:28:43.437415       1 snapshot_controller_base.go:358] updateSnapshot "default/test-rbd-snapshot"
>I1106 16:28:43.437431       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:28:43.437442       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:28:43.437445       1 snapshot_controller.go:182] syncSnapshot [default/test-rbd-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:28:43.437451       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:28:43.437455       1 snapshot_controller.go:191] syncSnapshot[default/test-rbd-snapshot]: check if we should add invalid label on snapshot
>I1106 16:28:43.437461       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:28:43.437466       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-rbd-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:28:43.437470       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:28:43.437477       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-rbd-snapshot
>I1106 16:28:43.437481       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-rbd-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:28:43.437485       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b].
>I1106 16:28:43.444887       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-cephfs-snapshot]
>I1106 16:28:43.444941       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:28:43.446335       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-rbd-snapshot]
>I1106 16:28:43.446356       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:28:43.451219       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:28:43.451244       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-cephfs-snapshot
>I1106 16:28:43.454722       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:28:43.454746       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-rbd-snapshot
>I1106 16:28:47.334879       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:28:52.346885       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:28:57.356821       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:29:02.369768       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:29:07.382544       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:29:12.393889       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:29:17.404053       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:29:22.414473       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:29:27.430114       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:29:32.442805       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:29:37.458462       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:29:42.468609       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:29:42.809011       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:29:42.809106       1 snapshot_controller_base.go:175] enqueued "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" for sync
>I1106 16:29:42.809123       1 snapshot_controller_base.go:175] enqueued "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" for sync
>I1106 16:29:42.809142       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:29:42.809162       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:29:42.809175       1 util.go:258] storeObjectUpdate updating content "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" with version 1565551
>I1106 16:29:42.809175       1 util.go:258] storeObjectUpdate updating content "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" with version 1565561
>I1106 16:29:42.809197       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: content is bound to snapshot default/test-cephfs-snapshot
>I1106 16:29:42.809198       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: content is bound to snapshot default/test-rbd-snapshot
>I1106 16:29:42.809203       1 snapshot_controller.go:88] syncContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: check if we should add invalid label on content
>I1106 16:29:42.809205       1 snapshot_controller.go:88] syncContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: check if we should add invalid label on content
>I1106 16:29:42.809211       1 snapshot_controller.go:1368] getSnapshotFromStore: snapshot default/test-rbd-snapshot found
>I1106 16:29:42.809219       1 snapshot_controller.go:989] needsUpdateSnapshotStatus[default/test-rbd-snapshot]
>I1106 16:29:42.809226       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [default/test-rbd-snapshot]: update snapshot status to true if needed.
>I1106 16:29:42.809211       1 snapshot_controller.go:1368] getSnapshotFromStore: snapshot default/test-cephfs-snapshot found
>I1106 16:29:42.809254       1 snapshot_controller.go:989] needsUpdateSnapshotStatus[default/test-cephfs-snapshot]
>I1106 16:29:42.809260       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [default/test-cephfs-snapshot]: update snapshot status to true if needed.
>I1106 16:29:42.809275       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-rbd-snapshot]
>I1106 16:29:42.809281       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-rbd-snapshot]
>I1106 16:29:42.809284       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-cephfs-snapshot]
>I1106 16:29:42.809290       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-cephfs-snapshot]
>I1106 16:29:42.809290       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-rbd-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>I1106 16:29:42.809296       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>I1106 16:29:42.809297       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-cephfs-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>I1106 16:29:42.809302       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>E1106 16:29:42.809311       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-cephfsplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-cephfsplugin-snapclass\" not found"
>E1106 16:29:42.809305       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-rbdplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-rbdplugin-snapclass\" not found"
>E1106 16:29:42.809361       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-rbdplugin-snapclass" not found
>E1106 16:29:42.809357       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-cephfsplugin-snapclass" not found
>I1106 16:29:42.809369       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-rbd-snapshot]
>I1106 16:29:42.809371       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-cephfs-snapshot]
>I1106 16:29:42.809374       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-rbd-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483dd0} is already set
>I1106 16:29:42.809418       1 snapshot_controller_base.go:220] Snapshot "default/test-rbd-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:29:42.809429       1 snapshot_controller_base.go:222] Updating snapshot "default/test-rbd-snapshot"
>I1106 16:29:42.809435       1 snapshot_controller_base.go:358] updateSnapshot "default/test-rbd-snapshot"
>I1106 16:29:42.809444       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:29:42.809455       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:29:42.809460       1 snapshot_controller.go:182] syncSnapshot [default/test-rbd-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:29:42.809473       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:29:42.809376       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-cephfs-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483d10} is already set
>I1106 16:29:42.809481       1 snapshot_controller.go:191] syncSnapshot[default/test-rbd-snapshot]: check if we should add invalid label on snapshot
>I1106 16:29:42.809489       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:29:42.809493       1 snapshot_controller_base.go:220] Snapshot "default/test-cephfs-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:29:42.809499       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-rbd-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:29:42.809514       1 snapshot_controller_base.go:222] Updating snapshot "default/test-cephfs-snapshot"
>I1106 16:29:42.809521       1 snapshot_controller_base.go:358] updateSnapshot "default/test-cephfs-snapshot"
>I1106 16:29:42.809523       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:29:42.809529       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:29:42.809536       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-rbd-snapshot
>I1106 16:29:42.809539       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:29:42.809543       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-rbd-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:29:42.809546       1 snapshot_controller.go:182] syncSnapshot [default/test-cephfs-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:29:42.809547       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b].
>I1106 16:29:42.809556       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:29:42.809568       1 snapshot_controller.go:191] syncSnapshot[default/test-cephfs-snapshot]: check if we should add invalid label on snapshot
>I1106 16:29:42.809576       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:29:42.809582       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-cephfs-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:29:42.809589       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:29:42.809603       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-cephfs-snapshot
>I1106 16:29:42.809610       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-cephfs-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:29:42.809616       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418].
>I1106 16:29:42.819589       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-rbd-snapshot]
>I1106 16:29:42.819754       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:29:42.821380       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-cephfs-snapshot]
>I1106 16:29:42.821414       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:29:42.832290       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:29:42.832315       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-rbd-snapshot
>I1106 16:29:42.837495       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:29:42.837524       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-cephfs-snapshot
>I1106 16:29:43.436563       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:29:43.436660       1 snapshot_controller_base.go:158] enqueued "default/test-rbd-snapshot" for sync
>I1106 16:29:43.437344       1 snapshot_controller_base.go:158] enqueued "default/test-cephfs-snapshot" for sync
>I1106 16:29:43.437368       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-rbd-snapshot]
>I1106 16:29:43.437374       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-rbd-snapshot]
>I1106 16:29:43.437382       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-rbd-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>I1106 16:29:43.437399       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>E1106 16:29:43.437408       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-rbdplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-rbdplugin-snapclass\" not found"
>E1106 16:29:43.437427       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-rbdplugin-snapclass" not found
>I1106 16:29:43.437435       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-rbd-snapshot]
>I1106 16:29:43.437439       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-rbd-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483dd0} is already set
>I1106 16:29:43.437492       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-cephfs-snapshot]
>I1106 16:29:43.437499       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-cephfs-snapshot]
>I1106 16:29:43.437514       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-cephfs-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>I1106 16:29:43.437518       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>E1106 16:29:43.437524       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-cephfsplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-cephfsplugin-snapclass\" not found"
>E1106 16:29:43.437530       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-cephfsplugin-snapclass" not found
>I1106 16:29:43.437539       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-cephfs-snapshot]
>I1106 16:29:43.437542       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-cephfs-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483d10} is already set
>I1106 16:29:43.437555       1 snapshot_controller_base.go:220] Snapshot "default/test-cephfs-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:29:43.437560       1 snapshot_controller_base.go:222] Updating snapshot "default/test-cephfs-snapshot"
>I1106 16:29:43.437564       1 snapshot_controller_base.go:358] updateSnapshot "default/test-cephfs-snapshot"
>I1106 16:29:43.437573       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:29:43.437583       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:29:43.437587       1 snapshot_controller.go:182] syncSnapshot [default/test-cephfs-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:29:43.437595       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:29:43.437601       1 snapshot_controller.go:191] syncSnapshot[default/test-cephfs-snapshot]: check if we should add invalid label on snapshot
>I1106 16:29:43.437609       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:29:43.437615       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-cephfs-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:29:43.437620       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:29:43.437631       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-cephfs-snapshot
>I1106 16:29:43.437641       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-cephfs-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:29:43.437646       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418].
>I1106 16:29:43.437945       1 snapshot_controller_base.go:220] Snapshot "default/test-rbd-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:29:43.437956       1 snapshot_controller_base.go:222] Updating snapshot "default/test-rbd-snapshot"
>I1106 16:29:43.437959       1 snapshot_controller_base.go:358] updateSnapshot "default/test-rbd-snapshot"
>I1106 16:29:43.437966       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:29:43.437973       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:29:43.437976       1 snapshot_controller.go:182] syncSnapshot [default/test-rbd-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:29:43.437984       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:29:43.437989       1 snapshot_controller.go:191] syncSnapshot[default/test-rbd-snapshot]: check if we should add invalid label on snapshot
>I1106 16:29:43.437994       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:29:43.437999       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-rbd-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:29:43.438012       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:29:43.438022       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-rbd-snapshot
>I1106 16:29:43.438027       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-rbd-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:29:43.438031       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b].
>I1106 16:29:43.448235       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-cephfs-snapshot]
>I1106 16:29:43.448384       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:29:43.449108       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-rbd-snapshot]
>I1106 16:29:43.449141       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:29:43.455744       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:29:43.455770       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-cephfs-snapshot
>I1106 16:29:43.463298       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:29:43.463407       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-rbd-snapshot
>I1106 16:29:47.479829       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:29:52.490569       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:29:57.504816       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:30:02.517628       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:30:07.527989       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:30:12.540023       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:30:17.551783       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:30:22.565523       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:30:27.576210       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:30:32.587265       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:30:37.603473       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:30:42.618694       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:30:42.809261       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:30:42.809556       1 snapshot_controller_base.go:175] enqueued "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" for sync
>I1106 16:30:42.809634       1 snapshot_controller_base.go:175] enqueued "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" for sync
>I1106 16:30:42.809706       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:30:42.809814       1 util.go:258] storeObjectUpdate updating content "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" with version 1565551
>I1106 16:30:42.809941       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: content is bound to snapshot default/test-cephfs-snapshot
>I1106 16:30:42.809981       1 snapshot_controller.go:88] syncContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: check if we should add invalid label on content
>I1106 16:30:42.810036       1 snapshot_controller.go:1368] getSnapshotFromStore: snapshot default/test-cephfs-snapshot found
>I1106 16:30:42.810072       1 snapshot_controller.go:989] needsUpdateSnapshotStatus[default/test-cephfs-snapshot]
>I1106 16:30:42.810169       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [default/test-cephfs-snapshot]: update snapshot status to true if needed.
>I1106 16:30:42.809767       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:30:42.810241       1 util.go:258] storeObjectUpdate updating content "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" with version 1565561
>I1106 16:30:42.810252       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: content is bound to snapshot default/test-rbd-snapshot
>I1106 16:30:42.810264       1 snapshot_controller.go:88] syncContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: check if we should add invalid label on content
>I1106 16:30:42.810277       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-cephfs-snapshot]
>I1106 16:30:42.810303       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-cephfs-snapshot]
>I1106 16:30:42.810320       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-cephfs-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>I1106 16:30:42.810334       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>E1106 16:30:42.810351       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-cephfsplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-cephfsplugin-snapclass\" not found"
>E1106 16:30:42.810374       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-cephfsplugin-snapclass" not found
>I1106 16:30:42.810393       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-cephfs-snapshot]
>I1106 16:30:42.810277       1 snapshot_controller.go:1368] getSnapshotFromStore: snapshot default/test-rbd-snapshot found
>I1106 16:30:42.810421       1 snapshot_controller.go:989] needsUpdateSnapshotStatus[default/test-rbd-snapshot]
>I1106 16:30:42.810425       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [default/test-rbd-snapshot]: update snapshot status to true if needed.
>I1106 16:30:42.810447       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-rbd-snapshot]
>I1106 16:30:42.810452       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-rbd-snapshot]
>I1106 16:30:42.810457       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-rbd-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>I1106 16:30:42.810460       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>E1106 16:30:42.810465       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-rbdplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-rbdplugin-snapclass\" not found"
>E1106 16:30:42.810474       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-rbdplugin-snapclass" not found
>I1106 16:30:42.810478       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-rbd-snapshot]
>I1106 16:30:42.810481       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-rbd-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483dd0} is already set
>I1106 16:30:42.810410       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-cephfs-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483d10} is already set
>I1106 16:30:42.810530       1 snapshot_controller_base.go:220] Snapshot "default/test-rbd-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:30:42.810537       1 snapshot_controller_base.go:222] Updating snapshot "default/test-rbd-snapshot"
>I1106 16:30:42.810542       1 snapshot_controller_base.go:358] updateSnapshot "default/test-rbd-snapshot"
>I1106 16:30:42.810551       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:30:42.810560       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:30:42.810566       1 snapshot_controller.go:182] syncSnapshot [default/test-rbd-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:30:42.810575       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:30:42.810581       1 snapshot_controller.go:191] syncSnapshot[default/test-rbd-snapshot]: check if we should add invalid label on snapshot
>I1106 16:30:42.810586       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:30:42.810590       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-rbd-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:30:42.810596       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:30:42.810604       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-rbd-snapshot
>I1106 16:30:42.810611       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-rbd-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:30:42.810615       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b].
>I1106 16:30:42.810624       1 snapshot_controller_base.go:220] Snapshot "default/test-cephfs-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:30:42.810638       1 snapshot_controller_base.go:222] Updating snapshot "default/test-cephfs-snapshot"
>I1106 16:30:42.810651       1 snapshot_controller_base.go:358] updateSnapshot "default/test-cephfs-snapshot"
>I1106 16:30:42.810666       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:30:42.810690       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:30:42.810708       1 snapshot_controller.go:182] syncSnapshot [default/test-cephfs-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:30:42.810724       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:30:42.810738       1 snapshot_controller.go:191] syncSnapshot[default/test-cephfs-snapshot]: check if we should add invalid label on snapshot
>I1106 16:30:42.810760       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:30:42.810777       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-cephfs-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:30:42.810790       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:30:42.810811       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-cephfs-snapshot
>I1106 16:30:42.810824       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-cephfs-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:30:42.810843       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418].
>I1106 16:30:42.821230       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-cephfs-snapshot]
>I1106 16:30:42.821355       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:30:42.822458       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-rbd-snapshot]
>I1106 16:30:42.822482       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:30:42.828923       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:30:42.829070       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-cephfs-snapshot
>I1106 16:30:42.830183       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:30:42.830200       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-rbd-snapshot
>I1106 16:30:43.436734       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:30:43.436908       1 snapshot_controller_base.go:158] enqueued "default/test-cephfs-snapshot" for sync
>I1106 16:30:43.436942       1 snapshot_controller_base.go:158] enqueued "default/test-rbd-snapshot" for sync
>I1106 16:30:43.436962       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-cephfs-snapshot]
>I1106 16:30:43.436976       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-cephfs-snapshot]
>I1106 16:30:43.436992       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-cephfs-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>I1106 16:30:43.437013       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-cephfsplugin-snapclass]
>E1106 16:30:43.437032       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-cephfsplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-cephfsplugin-snapclass\" not found"
>E1106 16:30:43.437052       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-cephfsplugin-snapclass" not found
>I1106 16:30:43.437070       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-cephfs-snapshot]
>I1106 16:30:43.437089       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-cephfs-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483d10} is already set
>I1106 16:30:43.437139       1 snapshot_controller_base.go:220] Snapshot "default/test-cephfs-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:30:43.437180       1 snapshot_controller_base.go:222] Updating snapshot "default/test-cephfs-snapshot"
>I1106 16:30:43.437194       1 snapshot_controller_base.go:358] updateSnapshot "default/test-cephfs-snapshot"
>I1106 16:30:43.437219       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:30:43.437239       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:30:43.437269       1 snapshot_controller.go:182] syncSnapshot [default/test-cephfs-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:30:43.437288       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:30:43.437323       1 snapshot_controller.go:191] syncSnapshot[default/test-cephfs-snapshot]: check if we should add invalid label on snapshot
>I1106 16:30:43.437346       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:30:43.437361       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-cephfs-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:30:43.437375       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-cephfs-snapshot]: bound to: "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418", Completed: false
>I1106 16:30:43.437394       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-cephfs-snapshot
>I1106 16:30:43.437408       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-cephfs-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:30:43.437430       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418].
>I1106 16:30:43.437812       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-rbd-snapshot]
>I1106 16:30:43.437864       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-rbd-snapshot]
>I1106 16:30:43.437881       1 snapshot_controller_base.go:328] checkAndUpdateSnapshotClass [test-rbd-snapshot]: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>I1106 16:30:43.437893       1 snapshot_controller.go:1176] getSnapshotClass: VolumeSnapshotClassName [ocs-storagecluster-rbdplugin-snapclass]
>E1106 16:30:43.437908       1 snapshot_controller.go:1180] failed to retrieve snapshot class ocs-storagecluster-rbdplugin-snapclass from the informer: "volumesnapshotclass.snapshot.storage.k8s.io \"ocs-storagecluster-rbdplugin-snapclass\" not found"
>E1106 16:30:43.437927       1 snapshot_controller_base.go:331] checkAndUpdateSnapshotClass failed to getSnapshotClass volumesnapshotclass.snapshot.storage.k8s.io "ocs-storagecluster-rbdplugin-snapclass" not found
>I1106 16:30:43.437953       1 snapshot_controller.go:721] updateSnapshotStatusWithEvent[default/test-rbd-snapshot]
>I1106 16:30:43.437965       1 snapshot_controller.go:724] updateSnapshotStatusWithEvent[test-rbd-snapshot]: the same error &{2020-11-06 15:13:28 +0000 UTC 0xc000483dd0} is already set
>I1106 16:30:43.437992       1 snapshot_controller_base.go:220] Snapshot "default/test-rbd-snapshot" is being deleted. SnapshotClass has already been removed
>I1106 16:30:43.438013       1 snapshot_controller_base.go:222] Updating snapshot "default/test-rbd-snapshot"
>I1106 16:30:43.438026       1 snapshot_controller_base.go:358] updateSnapshot "default/test-rbd-snapshot"
>I1106 16:30:43.438043       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:30:43.438060       1 snapshot_controller.go:180] synchronizing VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:30:43.438072       1 snapshot_controller.go:182] syncSnapshot [default/test-rbd-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
>I1106 16:30:43.438094       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:30:43.438127       1 snapshot_controller.go:191] syncSnapshot[default/test-rbd-snapshot]: check if we should add invalid label on snapshot
>I1106 16:30:43.438143       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:30:43.438164       1 snapshot_controller.go:272] processSnapshotWithDeletionTimestamp[default/test-rbd-snapshot]: delete snapshot content and remove finalizer from snapshot if needed
>I1106 16:30:43.438185       1 snapshot_controller.go:278] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[default/test-rbd-snapshot]: bound to: "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b", Completed: false
>I1106 16:30:43.438208       1 snapshot_controller.go:796] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot default/test-rbd-snapshot
>I1106 16:30:43.438225       1 snapshot_controller.go:297] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[default/test-rbd-snapshot]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:30:43.438252       1 snapshot_controller.go:308] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b].
>I1106 16:30:43.449676       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-rbd-snapshot]
>I1106 16:30:43.449712       1 snapshot_controller.go:897] cannot get claim from snapshot [test-rbd-snapshot]: [failed to retrieve PVC test-rbd from the lister: "persistentvolumeclaim \"test-rbd\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:30:43.450928       1 snapshot_controller.go:316] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[default/test-cephfs-snapshot]
>I1106 16:30:43.450954       1 snapshot_controller.go:897] cannot get claim from snapshot [test-cephfs-snapshot]: [failed to retrieve PVC test-cephfs from the lister: "persistentvolumeclaim \"test-cephfs\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
>I1106 16:30:43.461819       1 util.go:258] storeObjectUpdate updating snapshot "default/test-rbd-snapshot" with version 1565563
>I1106 16:30:43.461845       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-rbd-snapshot
>I1106 16:30:43.462236       1 util.go:258] storeObjectUpdate updating snapshot "default/test-cephfs-snapshot" with version 1565552
>I1106 16:30:43.462307       1 snapshot_controller.go:1344] Removed protection finalizer from volume snapshot default/test-cephfs-snapshot
>I1106 16:30:47.634310       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:30:52.646554       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:30:57.656927       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:31:02.668787       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:31:07.679277       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:31:12.689465       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:31:14.442683       1 snapshot_controller_base.go:158] enqueued "default/test-cephfs-snapshot" for sync
>I1106 16:31:14.442717       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-cephfs-snapshot]
>I1106 16:31:14.442722       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-cephfs-snapshot]
>I1106 16:31:14.442746       1 snapshot_controller_base.go:249] deleting snapshot "default/test-cephfs-snapshot"
>I1106 16:31:14.442752       1 snapshot_controller_base.go:409] snapshot "default/test-cephfs-snapshot" deleted
>I1106 16:31:14.442757       1 snapshot_controller_base.go:422] deleteSnapshot["default/test-cephfs-snapshot"]: scheduling sync of content snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418
>I1106 16:31:14.442783       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:31:14.442793       1 util.go:258] storeObjectUpdate updating content "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" with version 1565551
>I1106 16:31:14.442802       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: content is bound to snapshot default/test-cephfs-snapshot
>I1106 16:31:14.442806       1 snapshot_controller.go:88] syncContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: check if we should add invalid label on content
>I1106 16:31:14.442811       1 snapshot_controller.go:1359] getSnapshotFromStore: snapshot default/test-cephfs-snapshot not found
>I1106 16:31:17.699207       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:31:22.710445       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:31:27.723639       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:31:32.735823       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:31:37.745108       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:31:41.377094       1 snapshot_controller_base.go:158] enqueued "default/test-rbd-snapshot" for sync
>I1106 16:31:41.377122       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-rbd-snapshot]
>I1106 16:31:41.377127       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-rbd-snapshot]
>I1106 16:31:41.377142       1 snapshot_controller_base.go:249] deleting snapshot "default/test-rbd-snapshot"
>I1106 16:31:41.377149       1 snapshot_controller_base.go:409] snapshot "default/test-rbd-snapshot" deleted
>I1106 16:31:41.377156       1 snapshot_controller_base.go:422] deleteSnapshot["default/test-rbd-snapshot"]: scheduling sync of content snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b
>I1106 16:31:41.377307       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:31:41.377327       1 util.go:258] storeObjectUpdate updating content "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" with version 1565561
>I1106 16:31:41.377363       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: content is bound to snapshot default/test-rbd-snapshot
>I1106 16:31:41.377368       1 snapshot_controller.go:88] syncContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: check if we should add invalid label on content
>I1106 16:31:41.377373       1 snapshot_controller.go:1359] getSnapshotFromStore: snapshot default/test-rbd-snapshot not found
>I1106 16:31:42.757028       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:31:42.809636       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:31:42.809721       1 snapshot_controller_base.go:175] enqueued "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" for sync
>I1106 16:31:42.809735       1 snapshot_controller_base.go:175] enqueued "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" for sync
>I1106 16:31:42.809744       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:31:42.809754       1 util.go:258] storeObjectUpdate updating content "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" with version 1565561
>I1106 16:31:42.809768       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: content is bound to snapshot default/test-rbd-snapshot
>I1106 16:31:42.809771       1 snapshot_controller.go:88] syncContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: check if we should add invalid label on content
>I1106 16:31:42.809777       1 snapshot_controller.go:1359] getSnapshotFromStore: snapshot default/test-rbd-snapshot not found
>I1106 16:31:42.809778       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:31:42.809806       1 util.go:258] storeObjectUpdate updating content "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" with version 1565551
>I1106 16:31:42.809821       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: content is bound to snapshot default/test-cephfs-snapshot
>I1106 16:31:42.809839       1 snapshot_controller.go:88] syncContent[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]: check if we should add invalid label on content
>I1106 16:31:42.809845       1 snapshot_controller.go:1359] getSnapshotFromStore: snapshot default/test-cephfs-snapshot not found
>I1106 16:31:43.437125       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:31:47.767711       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:31:52.779889       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:31:57.789369       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:32:02.808096       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:32:07.819264       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:32:12.829653       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:32:17.842269       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:32:22.854461       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:32:27.865625       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:32:29.633691       1 snapshot_controller_base.go:175] enqueued "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" for sync
>I1106 16:32:29.633845       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418]
>I1106 16:32:29.633878       1 snapshot_controller_base.go:429] content "snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418" deleted
>I1106 16:32:29.633908       1 snapshot_controller_base.go:439] deleteContent["snapcontent-45ed1739-4e66-47ae-a725-9f754c8fc418"]: scheduling sync of snapshot default/test-cephfs-snapshot
>I1106 16:32:29.633948       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-cephfs-snapshot]
>I1106 16:32:29.633981       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-cephfs-snapshot]
>I1106 16:32:29.634003       1 snapshot_controller_base.go:240] deletion of snapshot "default/test-cephfs-snapshot" was already processed
>I1106 16:32:32.876254       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:32:37.885984       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:32:42.809819       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:32:42.810015       1 snapshot_controller_base.go:175] enqueued "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" for sync
>I1106 16:32:42.810062       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:32:42.810086       1 util.go:258] storeObjectUpdate updating content "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" with version 1565561
>I1106 16:32:42.810105       1 snapshot_controller.go:86] synchronizing VolumeSnapshotContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: content is bound to snapshot default/test-rbd-snapshot
>I1106 16:32:42.810126       1 snapshot_controller.go:88] syncContent[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]: check if we should add invalid label on content
>I1106 16:32:42.810141       1 snapshot_controller.go:1359] getSnapshotFromStore: snapshot default/test-rbd-snapshot not found
>I1106 16:32:42.899059       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:32:43.437303       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:32:46.022543       1 reflector.go:515] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PersistentVolumeClaim total 0 items received
>I1106 16:32:47.912843       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:32:49.805684       1 snapshot_controller_base.go:175] enqueued "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" for sync
>I1106 16:32:49.805822       1 snapshot_controller_base.go:277] syncContentByKey[snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b]
>I1106 16:32:49.805853       1 snapshot_controller_base.go:429] content "snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b" deleted
>I1106 16:32:49.805869       1 snapshot_controller_base.go:439] deleteContent["snapcontent-730a0fc5-18b5-441e-8b6f-2c448c23300b"]: scheduling sync of snapshot default/test-rbd-snapshot
>I1106 16:32:49.805907       1 snapshot_controller_base.go:202] syncSnapshotByKey[default/test-rbd-snapshot]
>I1106 16:32:49.805921       1 snapshot_controller_base.go:205] snapshotWorker: snapshot namespace [default] name [test-rbd-snapshot]
>I1106 16:32:49.805949       1 snapshot_controller_base.go:240] deletion of snapshot "default/test-rbd-snapshot" was already processed
>I1106 16:32:52.924622       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:32:57.937492       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:33:02.952359       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:33:07.961008       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:33:12.540897       1 reflector.go:515] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: Watch close - *v1beta1.VolumeSnapshotClass total 0 items received
>I1106 16:33:12.973546       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:33:18.000347       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:33:18.437451       1 reflector.go:515] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: Watch close - *v1beta1.VolumeSnapshot total 6 items received
>I1106 16:33:23.014577       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:33:28.026153       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:33:33.038641       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:33:36.810154       1 reflector.go:515] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: Watch close - *v1beta1.VolumeSnapshotContent total 5 items received
>I1106 16:33:38.049889       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:33:42.810132       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:33:43.063042       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:33:43.438020       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:33:48.070985       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:33:53.082535       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:33:58.095012       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:34:03.105426       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:34:08.115515       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:34:13.127867       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:34:18.145828       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:34:23.157952       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:34:28.169082       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:34:33.181889       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:34:38.196279       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:34:42.810355       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:34:43.240194       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:34:43.438203       1 reflector.go:369] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: forcing resync
>I1106 16:34:48.249547       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:34:53.260945       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:34:58.274636       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:35:03.288901       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:35:08.299316       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:35:13.310538       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader
>I1106 16:35:18.320651       1 leaderelection.go:273] successfully renewed lease openshift-cluster-storage-operator/snapshot-controller-leader

Actions: View

Attachments on bug 1893739: 1727165