Description of problem: CNV does not preallocate black image disks when requested. How reproducible: 100% Steps to Reproduce: 1. Create DV yaml with blank image disk and preallocation (see below). 2. oc create -f <file.yaml> Actual results: The storage file is not preallocated. Expected results: The storeage file should be preallocated. Additional info: Example dv YAML: ``` apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: blank-image-datavolume spec: source: blank: {} pvc: accessModes: - ReadWriteOnce resources: requests: storage: 500Mi preallocation: true ```
Moving the ball back into dev's court, there's no downstream release containing this fix yet.
Needed to support this use case.
Tomasz, can this be moved to ON_QA yet? Is it in a d/s build?
Tested with CDI v2.6.0-20 I requested 10Mi Ceph RBD, the storage file is not preallocated. I don't think OCS RBD space is full. [cnv-qe-jenkins@stg05-qixuan-b59vp-executor ~]$ oc logs -f importer-blank-image-datavolume-rbd I0201 15:08:26.299074 1 importer.go:52] Starting importer W0201 15:08:26.299246 1 importer.go:104] Available space less than requested size, creating blank image sized to available space: 8713216. I0201 15:08:26.299265 1 qemu.go:237] creating raw image with size 8713216, preallocation false I0201 15:08:26.311431 1 importer.go:213] Import Complete, Preallocation skipped
The quoted log proves that preallocation was processed correctly. If there's no available space on the block device, preallocation cannot be performed and must be skipped, otherwise the importer would fail. This case shows that the preallocation was tried and failed gracefully.
I noticed the bug title mentioned file data volumes, so I requested 10Mi CephFS space, the preallocation skipped either. Please correct me if I have some misunderstanding.
Do you have a filesystemOverhead setting set to a value bigger than 0? Maybe this is the problem. Quoting Alexander: "(...) if you create a 500Mi block device with a file system on it. The available space on the filesystem will be < 500Mi due to the file system overhead. So when calculating the size of the image we should take the overhead defined into account." You can try setting the overhead to, for example, 0.055: oc patch cdis.cdi.kubevirt.io cdi -p '{"spec": { "config": { "filesystemOverhead": {"global": "0.055"} }}}' -o json --type merge
After setting filesystemOverhead $ oc patch cdis.cdi.kubevirt.io cdi-kubevirt-hyperconverged -p '{"spec": { "config": { "filesystemOverhead": {"global": "0.055"} }}}' -o json --type merge A blank file data volume is preallocated. [cnv-qe-jenkins@stg05-qixuan-b59vp-executor ~]$ oc logs -f importer-blank-image-datavolume-cephfs I0202 14:31:51.038443 1 importer.go:52] Starting importer I0202 14:31:51.039701 1 qemu.go:237] creating raw image with size 100Mi, preallocation true I0202 14:31:51.039738 1 qemu.go:246] Added preallocation I0202 14:31:51.279703 1 importer.go:213] Import Complete, Preallocation applied [cnv-qe-jenkins@stg05-qixuan-b59vp-executor ~]$ oc get pvc blank-image-datavolume-cephfs -o yaml | grep preallo cdi.kubevirt.io/storage.preallocacation.requested: "true" cdi.kubevirt.io/storage.preallocation: "true" f:cdi.kubevirt.io/storage.preallocacation.requested: {} f:cdi.kubevirt.io/storage.preallocation: {} However, I'm confused about "if you create a 500Mi block device with a file system on it". Does that mean I can use Ceph RBD and create a DV with spec "volumeMode: Filesystem"? I got preallocation skipped. [cnv-qe-jenkins@stg05-qixuan-b59vp-executor ~]$ oc logs -f importer-blank-image-datavolume-rbd I0202 15:07:38.477940 1 importer.go:52] Starting importer W0202 15:07:38.478127 1 importer.go:104] Available space less than requested size, creating blank image sized to available space: 93648896. I0202 15:07:38.478143 1 qemu.go:237] creating raw image with size 93648896, preallocation false I0202 15:07:38.507634 1 importer.go:213] Import Complete, Preallocation skipped
In any case where the volume mode is filesystem you should be able to apply preallocation. If that is not working then we still have a bug here. Please share the DV spec, storage class spec, and importer logs.
DV spec [cnv-qe-jenkins@stg05-qixuan-b59vp-executor ~]$ cat blank-dv.yaml apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: blank-image-datavolume-rbd spec: source: blank: {} pvc: accessModes: - ReadWriteOnce resources: requests: storage: 500Mi storageClassName: ocs-storagecluster-ceph-rbd preallocation: true SC spec [cnv-qe-jenkins@stg05-qixuan-b59vp-executor ~]$ oc get sc ocs-storagecluster-ceph-rbd -o yaml allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: description: Provides RWO Filesystem volumes, and RWO and RWX Block volumes creationTimestamp: "2021-01-26T14:08:13Z" managedFields: - apiVersion: storage.k8s.io/v1 fieldsType: FieldsV1 fieldsV1: f:allowVolumeExpansion: {} f:metadata: f:annotations: .: {} f:description: {} f:parameters: .: {} f:clusterID: {} f:csi.storage.k8s.io/controller-expand-secret-name: {} f:csi.storage.k8s.io/controller-expand-secret-namespace: {} f:csi.storage.k8s.io/fstype: {} f:csi.storage.k8s.io/node-stage-secret-name: {} f:csi.storage.k8s.io/node-stage-secret-namespace: {} f:csi.storage.k8s.io/provisioner-secret-name: {} f:csi.storage.k8s.io/provisioner-secret-namespace: {} f:imageFeatures: {} f:imageFormat: {} f:pool: {} f:provisioner: {} f:reclaimPolicy: {} f:volumeBindingMode: {} manager: ocs-operator operation: Update time: "2021-01-26T14:08:13Z" name: ocs-storagecluster-ceph-rbd resourceVersion: "74616" selfLink: /apis/storage.k8s.io/v1/storageclasses/ocs-storagecluster-ceph-rbd uid: 983e5eb8-d455-4cca-b403-f7cd0b02d325 parameters: clusterID: openshift-storage csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage csi.storage.k8s.io/fstype: ext4 csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage imageFeatures: layering imageFormat: "2" pool: ocs-storagecluster-cephblockpool provisioner: openshift-storage.rbd.csi.ceph.com reclaimPolicy: Delete volumeBindingMode: Immediate Importer log [cnv-qe-jenkins@stg05-qixuan-b59vp-executor ~]$ oc logs -f importer-blank-image-datavolume-rbd I0203 03:19:49.468065 1 importer.go:52] Starting importer W0203 03:19:49.468235 1 importer.go:104] Available space less than requested size, creating blank image sized to available space: 492767232. I0203 03:19:49.468251 1 qemu.go:237] creating raw image with size 492767232, preallocation false I0203 03:19:49.484887 1 importer.go:213] Import Complete, Preallocation skipped
I think the code should preallocate to the adjusted size not just skip it. In this flow, the empty disk is created as large as CDI thinks is possible. Preallocation should not really care and just allocate the disk as it was created in the previous step. Tomasz, does that make sense to you or do you disagree?
Personally, I don't like computers trying to be smart ("Give me a 100M disk, please." – "Here's your 99M disk, sir" – "Whaaa? Where's the missing 1M?!"). On the other hand, trying to figure out what caused failure can be frustrating ("Give me a 100M disk, please." – "Can't do it, Dave." – "C'mon, give me the disk." – "Nope." – "Give it to me!" – "What's the magic word?"). So I can (grumpily) accept the "best effort" approach. In any case, let's open a new bug/story for that.
But this is the design. When creating a VM disk on a Filesystem mode PVC you will usually have less space than the PVC request size. Your compartmentalized preallocation feature should not worry about that logic as it's been already taken care of. The job of preallocation is simply to cause the already created disk to be fully allocated. I would prefer to continue tracking in this bug. Let's make sure we have a test case to check that we do not regress in this regard.
Pull request merged.
Setting back. Didn't notice that the pull request isn't a backport yet.
All the listed PRs are merged in release-v1.28. Waiting for it to reach downstream (it wasn't automatically updated because of a CI issue).
the v2.6.0-24 should include both PRs.
I can indeed see that the "skipped" path is removed (PRs 1651 and 1637) and I can preallocate blank Filesystem DVs on the virt-cdi-importer version that was mentioned. Keep in mind we are tracking Preallocation of Blank *Block* DVs at BZ 1928719. Verified on CNV 2.6.0, CDI: Containerized Data Importer v1.28.3-5-gde9eec6 HCO-v2.6.0-582 HCO image: registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:6b0e7bae456638acf8f8f5a8c0466f9b755627c961cb744e4ac0f3f5916793a9 CSV creation time: 2021-02-15 09:36:27 virt-cdi-importer 2.6.0-24
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 2.6.0 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:0799