Description of problem: When creating a VM from a template the cloned PVC status is Lost after attempting a smart clone and namespace transfer. Version-Release number of selected component (if applicable): virt-cdi-operator-container-v4.8.0-17 How reproducible: Always on my environment Steps to Reproduce: 1. Populate boot source for RHEL8 VM 2. Use Wizard to create VM from RHEL8 template Actual results: VM is pending due to PVC in Lost state Expected results: VM is ready to run with a Bound PVC Additional info: kind: PersistentVolumeClaim apiVersion: v1 metadata: annotations: pv.kubernetes.io/bind-completed: 'yes' volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com cdi.kubevirt.io/storage.populatedFor: rhel8-early-butterfly cdi.kubevirt.io/storage.condition.running.message: Clone Complete k8s.io/CloneOf: 'true' cdi.kubevirt.io/readyForTransfer: 'true' cdi.kubevirt.io/storage.condition.running: 'False' pv.kubernetes.io/bound-by-controller: 'yes' k8s.io/SmartCloneRequest: 'true' cdi.kubevirt.io/smartCloneSnapshot: >- openshift-virtualization-os-images/cdi-tmp-c6b796cf-c2c6-4a63-b625-485571e0b8cd cdi.kubevirt.io/storage.condition.running.reason: Completed cdi.kubevirt.io/ownedByDataVolume: default/rhel8-early-butterfly resourceVersion: '3892372' name: rhel8-early-butterfly uid: a6b576f8-126b-495f-a776-bc1d1bb46fba creationTimestamp: '2021-05-17T14:03:51Z' managedFields: - manager: kube-controller-manager operation: Update apiVersion: v1 time: '2021-05-17T14:03:51Z' fieldsType: FieldsV1 fieldsV1: 'f:status': 'f:phase': {} - manager: virt-cdi-controller operation: Update apiVersion: v1 time: '2021-05-17T14:03:51Z' fieldsType: FieldsV1 fieldsV1: 'f:metadata': 'f:annotations': 'f:cdi.kubevirt.io/storage.condition.running': {} 'f:k8s.io/CloneOf': {} 'f:cdi.kubevirt.io/readyForTransfer': {} 'f:cdi.kubevirt.io/ownedByDataVolume': {} .: {} 'f:pv.kubernetes.io/bound-by-controller': {} 'f:k8s.io/SmartCloneRequest': {} 'f:cdi.kubevirt.io/smartCloneSnapshot': {} 'f:cdi.kubevirt.io/storage.condition.running.reason': {} 'f:pv.kubernetes.io/bind-completed': {} 'f:volume.beta.kubernetes.io/storage-provisioner': {} 'f:cdi.kubevirt.io/storage.populatedFor': {} 'f:cdi.kubevirt.io/storage.condition.running.message': {} 'f:finalizers': .: {} 'v:"kubernetes.io/pvc-protection"': {} 'f:labels': .: {} 'f:app': {} 'f:cdi-controller': {} 'f:cdi.kubevirt.io': {} 'f:ownerReferences': .: {} 'k:{"uid":"c6b796cf-c2c6-4a63-b625-485571e0b8cd"}': .: {} 'f:apiVersion': {} 'f:blockOwnerDeletion': {} 'f:controller': {} 'f:kind': {} 'f:name': {} 'f:uid': {} 'f:spec': 'f:accessModes': {} 'f:dataSource': .: {} 'f:apiGroup': {} 'f:kind': {} 'f:name': {} 'f:resources': 'f:requests': .: {} 'f:storage': {} 'f:storageClassName': {} 'f:volumeMode': {} 'f:volumeName': {} namespace: default ownerReferences: - apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume name: rhel8-early-butterfly uid: c6b796cf-c2c6-4a63-b625-485571e0b8cd controller: true blockOwnerDeletion: true finalizers: - kubernetes.io/pvc-protection labels: app: containerized-data-importer cdi-controller: cdi-tmp-c6b796cf-c2c6-4a63-b625-485571e0b8cd cdi.kubevirt.io: cdi-smart-clone spec: accessModes: - ReadWriteMany resources: requests: storage: 20Gi volumeName: pvc-0187ee0c-01cb-4eaf-bc9f-204f3aca1c8c storageClassName: ocs-storagecluster-ceph-rbd volumeMode: Block dataSource: apiGroup: snapshot.storage.k8s.io kind: VolumeSnapshot name: cdi-tmp-c6b796cf-c2c6-4a63-b625-485571e0b8cd status: phase: Lost ========================================================================== kind: PersistentVolume apiVersion: v1 metadata: name: pvc-0187ee0c-01cb-4eaf-bc9f-204f3aca1c8c uid: 0522e176-915f-4f86-b322-01006635c37c resourceVersion: '3892421' creationTimestamp: '2021-05-17T14:03:50Z' annotations: pv.kubernetes.io/bound-by-controller: 'yes' pv.kubernetes.io/provisioned-by: openshift-storage.rbd.csi.ceph.com finalizers: - kubernetes.io/pv-protection managedFields: - manager: csi-provisioner operation: Update apiVersion: v1 time: '2021-05-17T14:03:50Z' fieldsType: FieldsV1 fieldsV1: 'f:metadata': 'f:annotations': .: {} 'f:pv.kubernetes.io/provisioned-by': {} 'f:spec': 'f:accessModes': {} 'f:capacity': .: {} 'f:storage': {} 'f:csi': .: {} 'f:controllerExpandSecretRef': .: {} 'f:name': {} 'f:namespace': {} 'f:driver': {} 'f:nodeStageSecretRef': .: {} 'f:name': {} 'f:namespace': {} 'f:volumeAttributes': 'f:clusterID': {} 'f:imageFeatures': {} .: {} 'f:storage.kubernetes.io/csiProvisionerIdentity': {} 'f:pool': {} 'f:imageName': {} 'f:radosNamespace': {} 'f:imageFormat': {} 'f:journalPool': {} 'f:volumeHandle': {} 'f:storageClassName': {} 'f:volumeMode': {} - manager: kube-controller-manager operation: Update apiVersion: v1 time: '2021-05-17T14:03:51Z' fieldsType: FieldsV1 fieldsV1: 'f:metadata': 'f:annotations': 'f:pv.kubernetes.io/bound-by-controller': {} 'f:spec': 'f:claimRef': .: {} 'f:apiVersion': {} 'f:kind': {} 'f:name': {} 'f:namespace': {} 'f:resourceVersion': {} 'f:uid': {} 'f:status': 'f:phase': {} - manager: virt-cdi-controller operation: Update apiVersion: v1 time: '2021-05-17T14:03:51Z' fieldsType: FieldsV1 fieldsV1: 'f:spec': 'f:persistentVolumeReclaimPolicy': {} spec: capacity: storage: 20Gi csi: driver: openshift-storage.rbd.csi.ceph.com volumeHandle: >- 0001-0011-openshift-storage-0000000000000001-b4a7a5bc-b718-11eb-9489-0a580a830019 volumeAttributes: clusterID: openshift-storage imageFeatures: layering imageFormat: '2' imageName: csi-vol-b4a7a5bc-b718-11eb-9489-0a580a830019 journalPool: ocs-storagecluster-cephblockpool pool: ocs-storagecluster-cephblockpool radosNamespace: '' storage.kubernetes.io/csiProvisionerIdentity: 1621015909261-8081-openshift-storage.rbd.csi.ceph.com nodeStageSecretRef: name: rook-csi-rbd-node namespace: openshift-storage controllerExpandSecretRef: name: rook-csi-rbd-provisioner namespace: openshift-storage accessModes: - ReadWriteMany claimRef: kind: PersistentVolumeClaim namespace: openshift-virtualization-os-images name: cdi-tmp-c6b796cf-c2c6-4a63-b625-485571e0b8cd uid: b189c6a2-7685-4526-8db8-cb68ada862ea apiVersion: v1 resourceVersion: '3892360' persistentVolumeReclaimPolicy: Retain storageClassName: ocs-storagecluster-ceph-rbd volumeMode: Block status: phase: Released ========================================================================== apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: annotations: cdi.kubevirt.io/cloneType: snapshot cdi.kubevirt.io/storage.clone.token: >- eyJhbGciOiJQUzI1NiIsImtpZCI6IiJ9.eyJleHAiOjE2MjEyNjA1MjcsImlhdCI6MTYyMTI2MDIyNywiaXNzIjoiY2RpLWFwaXNlcnZlciIsIm5hbWUiOiJyaGVsOCIsIm5hbWVzcGFjZSI6Im9wZW5zaGlmdC12aXJ0dWFsaXphdGlvbi1vcy1pbWFnZXMiLCJuYmYiOjE2MjEyNjAyMjcsIm9wZXJ0YXRpb24iOiJDbG9uZSIsInBhcmFtcyI6eyJ0YXJnZXROYW1lIjoicmhlbDgtZWFybHktYnV0dGVyZmx5IiwidGFyZ2V0TmFtZXNwYWNlIjoiZGVmYXVsdCJ9LCJyZXNvdXJjZSI6eyJncm91cCI6IiIsInJlc291cmNlIjoicGVyc2lzdGVudHZvbHVtZWNsYWltcyIsInZlcnNpb24iOiJ2MSJ9fQ.S1cZzXJS3rzci1MH9RRqc6J3gTmzZhDnxp1ctBg4UsdcRVDdFQinxGi3OmMN01XYYwN53f_LCmqIKMdFsuXMpaf_3vHmrDgNOTrbfTpmMr3g2AO6L5LBKa9pVAeCui5r2mKl5Qm2I32rRTMuZ2t0JoIWF3e_dwU4czLMbBhkh6WvRV4bnXd9NGfh0H1SFM8pbdJM8a76OJZMm5_eimLkhoZb3TLP4cFj-8wgw-k_6nsO_UNJL6fdR8oYFFE4HpZQdwpoQ-ghxHYN5B2b70FUBQFJp6HzAQin_rN56My_ME-F1p0N1dSht2f_ZvsFnawWiyP44CtGdNtVXz5lWiMetA resourceVersion: '3892339' name: rhel8-early-butterfly uid: c6b796cf-c2c6-4a63-b625-485571e0b8cd creationTimestamp: '2021-05-17T14:03:47Z' generation: 4 managedFields: - apiVersion: cdi.kubevirt.io/v1beta1 fieldsType: FieldsV1 fieldsV1: 'f:metadata': 'f:annotations': 'f:cdi.kubevirt.io/cloneType': {} 'f:finalizers': .: {} 'v:"cdi.kubevirt.io/dataVolumeFinalizer"': {} 'f:status': 'f:conditions': {} 'f:phase': {} manager: virt-cdi-controller operation: Update time: '2021-05-17T14:03:47Z' - apiVersion: cdi.kubevirt.io/v1alpha1 fieldsType: FieldsV1 fieldsV1: 'f:metadata': 'f:labels': .: {} 'f:kubevirt.io/created-by': {} 'f:ownerReferences': .: {} 'k:{"uid":"3b5d9c55-0471-4c5c-99f1-c9fe79706e53"}': .: {} 'f:apiVersion': {} 'f:blockOwnerDeletion': {} 'f:controller': {} 'f:kind': {} 'f:name': {} 'f:uid': {} 'f:spec': .: {} 'f:pvc': .: {} 'f:accessModes': {} 'f:resources': .: {} 'f:requests': .: {} 'f:storage': {} 'f:storageClassName': {} 'f:volumeMode': {} 'f:source': .: {} 'f:pvc': .: {} 'f:name': {} 'f:namespace': {} 'f:status': .: {} 'f:conditions': {} manager: virt-controller operation: Update time: '2021-05-17T14:03:47Z' namespace: default ownerReferences: - apiVersion: kubevirt.io/v1 blockOwnerDeletion: true controller: true kind: VirtualMachine name: rhel8-early-butterfly uid: 3b5d9c55-0471-4c5c-99f1-c9fe79706e53 finalizers: - cdi.kubevirt.io/dataVolumeFinalizer labels: kubevirt.io/created-by: 3b5d9c55-0471-4c5c-99f1-c9fe79706e53 spec: pvc: accessModes: - ReadWriteMany resources: requests: storage: 20Gi storageClassName: ocs-storagecluster-ceph-rbd volumeMode: Block source: pvc: name: rhel8 namespace: openshift-virtualization-os-images status: conditions: - lastHeartbeatTime: '2021-05-17T14:03:47Z' lastTransitionTime: '2021-05-17T14:03:47Z' message: No PVC found reason: NotFound status: Unknown type: Bound - lastHeartbeatTime: '2021-05-17T14:03:47Z' lastTransitionTime: '2021-05-17T14:03:47Z' status: 'False' type: Ready - lastHeartbeatTime: '2021-05-17T14:03:47Z' lastTransitionTime: '2021-05-17T14:03:47Z' status: 'False' type: Running phase: NamespaceTransferInProgress ========================================================================== kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: ocs-storagecluster-ceph-rbd uid: 8f5dc581-d428-4b6e-93e7-ef61483a210d resourceVersion: '203515' creationTimestamp: '2021-05-14T18:10:58Z' managedFields: - manager: ocs-operator operation: Update apiVersion: storage.k8s.io/v1 time: '2021-05-14T18:10:58Z' fieldsType: FieldsV1 fieldsV1: 'f:allowVolumeExpansion': {} 'f:parameters': 'f:csi.storage.k8s.io/controller-expand-secret-name': {} 'f:clusterID': {} 'f:imageFeatures': {} 'f:csi.storage.k8s.io/controller-expand-secret-namespace': {} 'f:csi.storage.k8s.io/fstype': {} .: {} 'f:csi.storage.k8s.io/provisioner-secret-namespace': {} 'f:pool': {} 'f:csi.storage.k8s.io/node-stage-secret-name': {} 'f:csi.storage.k8s.io/node-stage-secret-namespace': {} 'f:csi.storage.k8s.io/provisioner-secret-name': {} 'f:imageFormat': {} 'f:provisioner': {} 'f:reclaimPolicy': {} 'f:volumeBindingMode': {} provisioner: openshift-storage.rbd.csi.ceph.com parameters: csi.storage.k8s.io/fstype: ext4 csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner imageFormat: '2' clusterID: openshift-storage imageFeatures: layering csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage pool: ocs-storagecluster-cephblockpool csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage reclaimPolicy: Delete allowVolumeExpansion: true volumeBindingMode: Immediate ========================================================================== apiVersion: snapshot.storage.k8s.io/v1 deletionPolicy: Delete driver: openshift-storage.rbd.csi.ceph.com kind: VolumeSnapshotClass metadata: creationTimestamp: '2021-05-14T18:10:58Z' generation: 1 managedFields: - apiVersion: snapshot.storage.k8s.io/v1beta1 fieldsType: FieldsV1 fieldsV1: 'f:deletionPolicy': {} 'f:driver': {} 'f:parameters': .: {} 'f:clusterID': {} 'f:csi.storage.k8s.io/snapshotter-secret-name': {} 'f:csi.storage.k8s.io/snapshotter-secret-namespace': {} manager: ocs-operator operation: Update time: '2021-05-14T18:10:58Z' name: ocs-storagecluster-rbdplugin-snapclass resourceVersion: '203525' uid: 98126bbe-b27f-409e-a600-d98e6ec9c59c parameters: clusterID: openshift-storage csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/snapshotter-secret-namespace: openshift-storage ========================================================================== apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: annotations: cdi.kubevirt.io/ownedByDataVolume: default/rhel8-early-butterfly k8s.io/SmartCloneRequest: 'true' deletionTimestamp: '2021-05-17T14:03:50Z' resourceVersion: '3892328' name: cdi-tmp-c6b796cf-c2c6-4a63-b625-485571e0b8cd uid: 0687e2c8-2b6d-4b97-9de4-553c2e17718e deletionGracePeriodSeconds: 0 creationTimestamp: '2021-05-17T14:03:48Z' generation: 2 managedFields: - apiVersion: snapshot.storage.k8s.io/v1beta1 fieldsType: FieldsV1 fieldsV1: 'f:metadata': 'f:annotations': .: {} 'f:cdi.kubevirt.io/ownedByDataVolume': {} 'f:k8s.io/SmartCloneRequest': {} 'f:labels': .: {} 'f:app': {} 'f:cdi.kubevirt.io': {} 'f:spec': .: {} 'f:source': .: {} 'f:persistentVolumeClaimName': {} 'f:volumeSnapshotClassName': {} manager: virt-cdi-controller operation: Update time: '2021-05-17T14:03:48Z' - apiVersion: snapshot.storage.k8s.io/v1 fieldsType: FieldsV1 fieldsV1: 'f:metadata': 'f:finalizers': .: {} 'v:"snapshot.storage.kubernetes.io/volumesnapshot-bound-protection"': {} 'f:status': .: {} 'f:boundVolumeSnapshotContentName': {} 'f:creationTime': {} 'f:readyToUse': {} 'f:restoreSize': {} manager: snapshot-controller operation: Update time: '2021-05-17T14:03:50Z' namespace: openshift-virtualization-os-images finalizers: - snapshot.storage.kubernetes.io/volumesnapshot-bound-protection labels: app: containerized-data-importer cdi.kubevirt.io: cdi-smart-clone spec: source: persistentVolumeClaimName: rhel8 volumeSnapshotClassName: ocs-storagecluster-rbdplugin-snapclass status: boundVolumeSnapshotContentName: snapcontent-0687e2c8-2b6d-4b97-9de4-553c2e17718e creationTime: '2021-05-17T14:03:49Z' readyToUse: true restoreSize: 20Gi ========================================================================== apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotContent metadata: annotations: snapshot.storage.kubernetes.io/deletion-secret-name: rook-csi-rbd-provisioner snapshot.storage.kubernetes.io/deletion-secret-namespace: openshift-storage snapshot.storage.kubernetes.io/volumesnapshot-being-deleted: 'yes' deletionTimestamp: '2021-05-17T14:03:50Z' resourceVersion: '3892327' name: snapcontent-0687e2c8-2b6d-4b97-9de4-553c2e17718e uid: c01bb1ad-bbb6-4cd0-a79c-39e9b0972a71 deletionGracePeriodSeconds: 0 creationTimestamp: '2021-05-17T14:03:48Z' generation: 2 managedFields: - apiVersion: snapshot.storage.k8s.io/v1beta1 fieldsType: FieldsV1 fieldsV1: 'f:status': .: {} 'f:creationTime': {} 'f:readyToUse': {} 'f:restoreSize': {} 'f:snapshotHandle': {} manager: csi-snapshotter operation: Update time: '2021-05-17T14:03:50Z' - apiVersion: snapshot.storage.k8s.io/v1 fieldsType: FieldsV1 fieldsV1: 'f:metadata': 'f:annotations': .: {} 'f:snapshot.storage.kubernetes.io/deletion-secret-name': {} 'f:snapshot.storage.kubernetes.io/deletion-secret-namespace': {} 'f:snapshot.storage.kubernetes.io/volumesnapshot-being-deleted': {} 'f:finalizers': .: {} 'v:"snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection"': {} 'f:spec': .: {} 'f:deletionPolicy': {} 'f:driver': {} 'f:source': .: {} 'f:volumeHandle': {} 'f:volumeSnapshotClassName': {} 'f:volumeSnapshotRef': .: {} 'f:apiVersion': {} 'f:kind': {} 'f:name': {} 'f:namespace': {} 'f:resourceVersion': {} 'f:uid': {} manager: snapshot-controller operation: Update time: '2021-05-17T14:03:50Z' finalizers: - snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection spec: deletionPolicy: Delete driver: openshift-storage.rbd.csi.ceph.com source: volumeHandle: >- 0001-0011-openshift-storage-0000000000000001-2316c1b8-b4e7-11eb-9489-0a580a830019 volumeSnapshotClassName: ocs-storagecluster-rbdplugin-snapclass volumeSnapshotRef: apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot name: cdi-tmp-c6b796cf-c2c6-4a63-b625-485571e0b8cd namespace: openshift-virtualization-os-images resourceVersion: '3892251' uid: 0687e2c8-2b6d-4b97-9de4-553c2e17718e status: creationTime: 1621260229049759200 readyToUse: true restoreSize: 21474836480 snapshotHandle: >- 0001-0011-openshift-storage-0000000000000001-b3a0d5dc-b718-11eb-9489-0a580a830019
Created attachment 1784096 [details] Lost PVC in UI
Michael can you please take a look?
Created attachment 1784097 [details] cdi-deployment logs
I see what's going on. Coordination between the datavolumecontroller and the smartclone controller is broken when it takes awhile to delete a snapshot
Peter, This should be considered a release blocker because this bug interferes with the smart clone flows. Can you please approve?
Verified with the following code: ------------------------------------------------------ oc version Client Version: 4.8.0-fc.5 Server Version: 4.8.0-fc.5 Kubernetes Version: v1.21.0-rc.0+88a3e8c [cnv-qe-jenkins@stg03-kevin-zwzbq-executor kevin]$ oc get csv -n openshift-cnv NAME DISPLAY VERSION REPLACES PHASE kubevirt-hyperconverged-operator.v4.8.0 OpenShift Virtualization 4.8.0 kubevirt-hyperconverged-operator.v2.6.2 Succeeded Verified with the following scenario: ------------------------------------------------------ Steps to Reproduce: 1. Populate boot source for RHEL8 VM 2. Use Wizard to create VM from RHEL8 template oc get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE openshift-cnv rhel8-short-iguana Bound pvc-6e51cf8d-f950-4664-bdc7-7ad30ff38f67 20Gi RWX ocs-storagecluster-ceph-rbd 73s VM is ready to run with a Bound PVC Moving to VERIFIED!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.8.0 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2920