Bug 1914177 - CNV does not preallocate blank file data volumes
Summary: CNV does not preallocate blank file data volumes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Storage
Version: 2.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 2.6.0
Assignee: Tomasz Barański
QA Contact: Alex Kalenyuk
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-08 09:52 UTC by Tomasz Barański
Modified: 2021-03-10 11:23 UTC (History)
7 users (show)

Fixed In Version: virt-cdi-importer v2.6.0-24
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-10 11:22:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt containerized-data-importer pull 1555 0 None closed core: Preallocate blank image disks as well 2021-02-15 02:04:59 UTC
Github kubevirt containerized-data-importer pull 1561 0 None closed core: Preallocate blank image disks as well 2021-02-15 02:04:59 UTC
Github kubevirt containerized-data-importer pull 1637 0 None closed Preallocate even if the size is too small 2021-02-14 12:10:35 UTC
Github kubevirt containerized-data-importer pull 1651 0 None closed [release-v1.28] Preallocate even if the size is too small 2021-02-15 15:22:57 UTC
Red Hat Product Errata RHSA-2021:0799 0 None None None 2021-03-10 11:23:17 UTC

Description Tomasz Barański 2021-01-08 09:52:28 UTC
Description of problem:

CNV does not preallocate black image disks when requested.

How reproducible:
100%

Steps to Reproduce:
1. Create DV yaml with blank image disk and preallocation (see below).
2. oc create -f <file.yaml>

Actual results:
The storage file is not preallocated.

Expected results:
The storeage file should be preallocated.


Additional info:

Example dv YAML:

```
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: blank-image-datavolume
spec:
  source:
      blank: {}
  pvc:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 500Mi
  preallocation: true
```

Comment 1 Maya Rashish 2021-01-12 13:21:38 UTC
Moving the ball back into dev's court, there's no downstream release containing this fix yet.

Comment 2 Peter Lauterbach 2021-01-12 14:59:50 UTC
Needed to support this use case.

Comment 3 Adam Litke 2021-01-25 14:34:40 UTC
Tomasz, can this be moved to ON_QA yet?  Is it in a d/s build?

Comment 4 Qixuan Wang 2021-02-01 16:11:44 UTC
Tested with CDI v2.6.0-20

I requested 10Mi Ceph RBD, the storage file is not preallocated. I don't think OCS RBD space is full.

[cnv-qe-jenkins@stg05-qixuan-b59vp-executor ~]$ oc logs -f importer-blank-image-datavolume-rbd
I0201 15:08:26.299074       1 importer.go:52] Starting importer
W0201 15:08:26.299246       1 importer.go:104] Available space less than requested size, creating blank image sized to available space: 8713216.
I0201 15:08:26.299265       1 qemu.go:237] creating raw image with size 8713216, preallocation false
I0201 15:08:26.311431       1 importer.go:213] Import Complete, Preallocation skipped

Comment 5 Tomasz Barański 2021-02-02 09:24:47 UTC
The quoted log proves that preallocation was processed correctly. If there's no available space on the block device, preallocation cannot be performed and must be skipped, otherwise the importer would fail. This case shows that the preallocation was tried and failed gracefully.

Comment 7 Qixuan Wang 2021-02-02 11:26:27 UTC
I noticed the bug title mentioned file data volumes, so I requested 10Mi CephFS space, the preallocation skipped either. Please correct me if I have some misunderstanding.

Comment 8 Tomasz Barański 2021-02-02 14:00:13 UTC
Do you have a filesystemOverhead setting set to a value bigger than 0? Maybe this is the problem. Quoting Alexander: "(...) if you create a 500Mi block device with a file system on it. The available space on the filesystem will be < 500Mi due to the file system overhead. So when calculating the size of the  image we should take the overhead defined into account."

You can try setting the overhead to, for example, 0.055:

    oc patch cdis.cdi.kubevirt.io cdi -p '{"spec": { "config": { "filesystemOverhead": {"global": "0.055"} }}}' -o json --type merge

Comment 9 Qixuan Wang 2021-02-02 15:18:04 UTC
After setting filesystemOverhead
$ oc patch cdis.cdi.kubevirt.io cdi-kubevirt-hyperconverged -p '{"spec": { "config": { "filesystemOverhead": {"global": "0.055"} }}}' -o json --type merge

A blank file data volume is preallocated.
[cnv-qe-jenkins@stg05-qixuan-b59vp-executor ~]$ oc logs -f importer-blank-image-datavolume-cephfs
I0202 14:31:51.038443       1 importer.go:52] Starting importer
I0202 14:31:51.039701       1 qemu.go:237] creating raw image with size 100Mi, preallocation true
I0202 14:31:51.039738       1 qemu.go:246] Added preallocation
I0202 14:31:51.279703       1 importer.go:213] Import Complete, Preallocation applied

[cnv-qe-jenkins@stg05-qixuan-b59vp-executor ~]$ oc get pvc blank-image-datavolume-cephfs -o yaml | grep preallo
    cdi.kubevirt.io/storage.preallocacation.requested: "true"
    cdi.kubevirt.io/storage.preallocation: "true"
          f:cdi.kubevirt.io/storage.preallocacation.requested: {}
          f:cdi.kubevirt.io/storage.preallocation: {}

However, I'm confused about "if you create a 500Mi block device with a file system on it". Does that mean I can use Ceph RBD and create a DV with spec "volumeMode: Filesystem"? I got preallocation skipped.
[cnv-qe-jenkins@stg05-qixuan-b59vp-executor ~]$ oc logs -f importer-blank-image-datavolume-rbd
I0202 15:07:38.477940       1 importer.go:52] Starting importer
W0202 15:07:38.478127       1 importer.go:104] Available space less than requested size, creating blank image sized to available space: 93648896.
I0202 15:07:38.478143       1 qemu.go:237] creating raw image with size 93648896, preallocation false
I0202 15:07:38.507634       1 importer.go:213] Import Complete, Preallocation skipped

Comment 10 Adam Litke 2021-02-02 18:11:16 UTC
In any case where the volume mode is filesystem you should be able to apply preallocation.  If that is not working then we still have a bug here.  Please share the DV spec, storage class spec, and importer logs.

Comment 11 Qixuan Wang 2021-02-03 03:22:53 UTC
DV spec
[cnv-qe-jenkins@stg05-qixuan-b59vp-executor ~]$ cat blank-dv.yaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: blank-image-datavolume-rbd
spec:
  source:
      blank: {}
  pvc:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 500Mi
    storageClassName: ocs-storagecluster-ceph-rbd
  preallocation: true



SC spec
[cnv-qe-jenkins@stg05-qixuan-b59vp-executor ~]$ oc get sc ocs-storagecluster-ceph-rbd -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    description: Provides RWO Filesystem volumes, and RWO and RWX Block volumes
  creationTimestamp: "2021-01-26T14:08:13Z"
  managedFields:
  - apiVersion: storage.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:allowVolumeExpansion: {}
      f:metadata:
        f:annotations:
          .: {}
          f:description: {}
      f:parameters:
        .: {}
        f:clusterID: {}
        f:csi.storage.k8s.io/controller-expand-secret-name: {}
        f:csi.storage.k8s.io/controller-expand-secret-namespace: {}
        f:csi.storage.k8s.io/fstype: {}
        f:csi.storage.k8s.io/node-stage-secret-name: {}
        f:csi.storage.k8s.io/node-stage-secret-namespace: {}
        f:csi.storage.k8s.io/provisioner-secret-name: {}
        f:csi.storage.k8s.io/provisioner-secret-namespace: {}
        f:imageFeatures: {}
        f:imageFormat: {}
        f:pool: {}
      f:provisioner: {}
      f:reclaimPolicy: {}
      f:volumeBindingMode: {}
    manager: ocs-operator
    operation: Update
    time: "2021-01-26T14:08:13Z"
  name: ocs-storagecluster-ceph-rbd
  resourceVersion: "74616"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/ocs-storagecluster-ceph-rbd
  uid: 983e5eb8-d455-4cca-b403-f7cd0b02d325
parameters:
  clusterID: openshift-storage
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
  imageFeatures: layering
  imageFormat: "2"
  pool: ocs-storagecluster-cephblockpool
provisioner: openshift-storage.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate


Importer log
[cnv-qe-jenkins@stg05-qixuan-b59vp-executor ~]$ oc logs -f importer-blank-image-datavolume-rbd
I0203 03:19:49.468065       1 importer.go:52] Starting importer
W0203 03:19:49.468235       1 importer.go:104] Available space less than requested size, creating blank image sized to available space: 492767232.
I0203 03:19:49.468251       1 qemu.go:237] creating raw image with size 492767232, preallocation false
I0203 03:19:49.484887       1 importer.go:213] Import Complete, Preallocation skipped

Comment 14 Adam Litke 2021-02-03 17:49:30 UTC
I think the code should preallocate to the adjusted size not just skip it.  In this flow, the empty disk is created as large as CDI thinks is possible.  Preallocation should not really care and just allocate the disk as it was created in the previous step.

Tomasz, does that make sense to you or do you disagree?

Comment 15 Tomasz Barański 2021-02-04 10:43:38 UTC
Personally, I don't like computers trying to be smart ("Give me a 100M disk, please." – "Here's your 99M disk, sir" – "Whaaa? Where's the missing 1M?!"). On the other hand, trying to figure out what caused failure can be frustrating ("Give  me a 100M disk, please." – "Can't do it, Dave." – "C'mon, give me the disk." – "Nope." – "Give it to me!" – "What's the magic word?"). So I can (grumpily) accept the "best effort" approach.

In any case, let's open a new bug/story for that.

Comment 16 Adam Litke 2021-02-04 13:09:46 UTC
But this is the design.  When creating a VM disk on a Filesystem mode PVC you will usually have less space than the PVC request size.  Your compartmentalized preallocation feature should not worry about that logic as it's been already taken care of.  The job of preallocation is simply to cause the already created disk to be fully allocated.

I would prefer to continue tracking in this bug.  Let's make sure we have a test case to check that we do not regress in this regard.

Comment 17 Maya Rashish 2021-02-11 07:22:24 UTC
Pull request merged.

Comment 18 Maya Rashish 2021-02-11 07:25:04 UTC
Setting back. Didn't notice that the pull request isn't a backport yet.

Comment 19 Maya Rashish 2021-02-14 12:14:39 UTC
All the listed PRs are merged in release-v1.28.
Waiting for it to reach downstream (it wasn't automatically updated because of a CI issue).

Comment 20 Maya Rashish 2021-02-15 07:53:58 UTC
the v2.6.0-24 should include both PRs.

Comment 21 Alex Kalenyuk 2021-02-15 14:36:36 UTC
I can indeed see that the "skipped" path is removed (PRs 1651 and 1637) and I can preallocate blank Filesystem DVs on the virt-cdi-importer version that was mentioned.
Keep in mind we are tracking Preallocation of Blank *Block* DVs at BZ 1928719.

Verified on CNV 2.6.0, CDI: Containerized Data Importer v1.28.3-5-gde9eec6
HCO-v2.6.0-582
HCO image: registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:6b0e7bae456638acf8f8f5a8c0466f9b755627c961cb744e4ac0f3f5916793a9
CSV creation time: 2021-02-15 09:36:27
virt-cdi-importer 2.6.0-24

Comment 24 errata-xmlrpc 2021-03-10 11:22:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 2.6.0 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0799


Note You need to log in before you can comment on or make changes to this bug.