Bug 2040300

Created attachment 1850577 [details]
Description of the DV, VM, PVC, and PV:

Description of problem:
Clone of HPP VM to OCS Block VM gets stuck in 100.00% progress and restarts, log says "no space left on device"

Version-Release number of selected component (if applicable):
4.10

How reproducible:
Always

Steps to Reproduce:
1. Create an HPP VM.
2. Stop the VM if it's running (virtctl stop <vm-cirros-hpp>).
3. Clone HPP VM to OCS Block VM.

Actual results:
CloneInProgress 100.00%, restarts over and over.

$ oc get dv -A
NAMESPACE                            NAME                   PHASE              PROGRESS   RESTARTS   AGE
default                              dv-cirros-hpp          Succeeded          100.0%                9m26s
default                              target-dv-vm-clone     CloneInProgress    0.11%      2          7m33s
openshift-virtualization-os-images   centos8-4a6a0eee7b4a   ImportInProgress   N/A        229        19h
openshift-virtualization-os-images   fedora-3b3fc310abea    ImportInProgress   N/A        229        19h
openshift-virtualization-os-images   rhel8-2325b0922721     Succeeded          100.0%                19h
openshift-virtualization-os-images   rhel9-401cc6250f89     Succeeded          100.0%                19h

$ oc get pods -A | grep target-dv
default                                            cdi-upload-target-dv-vm-clone                                     1/1     Running            0                 12m

$ oc logs cdi-upload-target-dv-vm-clone
I0113 10:31:33.293251       1 uploadserver.go:69] Running server on 0.0.0.0:8443
I0113 10:31:47.569368       1 uploadserver.go:374] Content type header is "filesystem-clone"
I0113 10:31:47.583319       1 uploadserver.go:477] Untaring 248598075904 bytes to /dev/cdi-block-volume
E0113 10:34:33.920053       1 uploadserver.go:387] Saving stream failed: error unarchiving to /dev/cdi-block-volume: write /dev/cdi-block-volume: no space left on device
I0113 10:34:35.200854       1 uploadserver.go:374] Content type header is "filesystem-clone"
I0113 10:34:35.223046       1 uploadserver.go:477] Untaring 248598075904 bytes to /dev/cdi-block-volume
E0113 10:37:21.465909       1 uploadserver.go:387] Saving stream failed: error unarchiving to /dev/cdi-block-volume: write /dev/cdi-block-volume: no space left on device
I0113 10:37:38.141314       1 uploadserver.go:374] Content type header is "filesystem-clone"
I0113 10:37:38.207757       1 uploadserver.go:477] Untaring 248598075904 bytes to /dev/cdi-block-volume
E0113 10:40:23.607053       1 uploadserver.go:387] Saving stream failed: error unarchiving to /dev/cdi-block-volume: write /dev/cdi-block-volume: no space left on device
I0113 10:40:54.716272       1 uploadserver.go:374] Content type header is "filesystem-clone"
I0113 10:40:54.728761       1 uploadserver.go:477] Untaring 248598075904 bytes to /dev/cdi-block-volume

Expected results:
Clone Succeeded


Additional info:

Clone HPP VM to NFS VM - Succeeded.
Clone HPP DV (DV was created separately, without the VM) to OCS Block VM - Succeeded.

-----------------------------

YAMLS to reproduce the bug:

$ cat vm-hpp-cirros.yaml 
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  creationTimestamp: null
  labels:
    kubevirt.io/vm: vm-cirros-hpp
  name: vm-cirros-hpp
spec:
  dataVolumeTemplates:
  - metadata:
      creationTimestamp: null
      name: dv-cirros-hpp
    spec:
      pvc:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: hostpath-provisioner
      source:
        http:
          url: http://.../cirros-images/cirros-0.4.0-x86_64-disk.qcow2
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/vm: vm-cirros-hpp
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: datavolume-cirros-hpp
        machine:
          type: ""
        resources:
          requests:
            memory: 100Mi
      terminationGracePeriodSeconds: 0
      volumes:
      - dataVolume:
          name: dv-cirros-hpp
        name: datavolume-cirros-hpp


$ cat vm-ocs-block-cirros.yaml 
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: vm-dv-clone
  name: vm-dv-clone
spec:
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/vm: vm-dv-clone
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: root-disk
        resources:
          requests:
            memory: 100Mi
      volumes:
      - dataVolume:
          name: target-dv-vm-clone
        name: root-disk
  dataVolumeTemplates:
  - metadata:
      name: target-dv-vm-clone
    spec:
      pvc:
        storageClassName: ocs-storagecluster-ceph-rbd
        volumeMode: Block
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 15Gi
      source:
        pvc:
          namespace: "default"
          name: "dv-cirros-hpp"