Description of problem: ----------------------- VM Migration to Ceph RBD block storage fails with this error. DataVolumeCreationFailed: Error while importing disk image: mini-rhel7-cloudinit-harddisk1. pod CrashLoopBackoff restart exceeded Also, see https://bugzilla.redhat.com/show_bug.cgi?id=1898478 Version-Release number of selected component (if applicable): ------------------------------------------------------------ MTV 2.0 How reproducible: ---------------- Always Steps to Reproduce: ------------------- 1.Migrate a VM from VMware to OpenshiftV using API 2.Target storage should be Ceph RBD block storage 3.In the migration plan, set the Volume mode to 'Block' for the destination datastore. Actual results: -------------- Migration fails because of a data volume creation failure. Expected results: ----------------- Data volume creation should succeed; migration should succeed. Additional info: -----------------
This issue reproduces. Migration plan ============== $ cat << EOF | oc apply -f - --- apiVersion: forklift.konveyor.io/v1alpha1 kind: Plan metadata: name: plan-no-volmode namespace: openshift-migration spec: provider: source: name: vmware1 namespace: openshift-migration destination: name: host namespace: openshift-migration map: networks: - source: name: VM Network destination: type: pod name: pod namespace: openshift-migration datastores: - source: name: rhv-v2v-performance-testing destination: storageClass: ocs-storagecluster-ceph-rbd vms: - name: mini-rhel7-istein EOF VM import fail with this error in CNV UI: ======================================== "Import error (RHV) mini-rhel7-istein could not be imported. DataVolumeCreationFailed: Error while importing disk image: mini-rhel7-istein-harddisk1. pod CrashLoopBackoff restart exceeded" vm-import-controller.log error: ============================== level":"info","ts":1608730612.7150602,"logger":"controller_virtualmachineimport","msg":"CDI import pod failed.","Request.Namespace":"openshift-migration","Request.Name":"plan-plan-no-volmode-vm-689-cd18","VM.Name":{"namespace":"openshift-migration","name":"mini-rhel7-istein"}} importer pod error: ================== E1223 13:34:43.891962 1 prlimit.go:176] qemu-img: /data/disk.img: error while converting raw: Could not create file: No such file or directory I1223 13:34:43.892001 1 vddk-datasource.go:249] Failed to convert disk: could not stream/convert image to raw: qemu-img execution failed: exit status 1
Created attachment 1741558 [details] importer-pod.log
Created attachment 1741573 [details] vm-import-controller.log
Created attachment 1741574 [details] oc_describe_dv.yaml
This bug is specific to VM import from VMware using VMIO, based on these tests: 1. VM import from VMware via api using cdi importer - Fail, same as MTV migration failed (details in comment #2) 2. VM import from RHV via api to ceph-rbd/block using cdi importer - Pass 3. VM import from VMware (same VM from 1) via UI (NOT using cdi importer) - Pass for all 3, Target storage: Ceph-RBD, volumeMode:Block, AccessMode: ReadWriteOnly. In CNV-2.5.0, VMware to CNV VM import using VMIO worked OK or Ceph-RBD/Block. Therefore this is a regression. Moving bug to CNV/v2v.
Adding that comments #2 and #6 refer to CNV-2.5.3.
Would it be possible to have the YAML of the importer pod ? And even access to the cluster where it fails, with a reproducer plan ? Note that `oc get -o yaml` is preferred to `oc describe` because it provides that raw YAML of the object, while `oc describe` tries to format it somehow and may lose / hide some information.
I've changed the component to Storage, because the error happens in CDI, not VMIO. Would it be possible to have the YAML (via "oc get -o yaml") for the DataVolume, too?
Created attachment 1743249 [details] oc_get_importer_pod-oyaml
Created attachment 1743250 [details] oc_get_dv-oyaml
Alexander, can you take a look. I'm wondering if the combination of VMWare and block mode is problematic.
The pod definition looks fine at first glance, the block device is properly added to it. But for some reason it thinks it is using a file system as it is trying to write to /data/disk.img which it shouldn't. It should be attempting to write to the block device. I will look into it more closely once I get done with some other stuff.
I investigated the issue, and the problem got introduced here [0]. destinationFile is hard coded to /data/disk.img, while it should be fileName which gets calculated based on the type of volume. In newer version of the code all of this has been modified, do we want to fix this in the 2.5 stable branch? [0] https://github.com/kubevirt/containerized-data-importer/pull/1320/files#diff-c46055929ede7b3ba517257f9a8614ee9507c33a99c80b8d51933e143c6fb864R247
@fdupont Is it okay to target this to 2.6 or do you need something for 2.5 still?
Verified on CNV-2.6: hco-bundle-registry-container-v2.6.0-520. VM import from VMware via api of a RHEL7 VM to target Ceph-RBD/Block ended successfully and VM was started OK. (Tested with "default" namespace). Based on that move to Verified.
The verification in comment #20 is based on https://bugzilla.redhat.com/show_bug.cgi?id=1917908#c9 Another VM import using ceph-RBD block bug that we had on CNV-2.6.0 and resolved.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 2.6.0 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:0799