This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
Bug 2237287 - portworx: disk.img size is smaller than requested
Summary: portworx: disk.img size is smaller than requested
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Storage
Version: 4.14.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.14.2
Assignee: Alex Kalenyuk
QA Contact: Jenia Peimer
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-09-04 14:31 UTC by Jenia Peimer
Modified: 2023-12-14 16:11 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
Cause: Due to double accounting of filesystem overhead with Portworx storage Consequence: in some cases a VM disk image may be created with a size smaller than intended. Workaround (if any): The PVC may be expanded to add additional space after initial provisioning is completed. Result: Extra space can be made available to correct the issue.
Clone Of:
Environment:
Last Closed: 2023-12-14 16:11:37 UTC
Target Upstream Version:
Embargoed:
pousley: needinfo-
pousley: needinfo-

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker   CNV-32695 0 None None None 2023-12-14 16:11:36 UTC

Description Jenia Peimer 2023-09-04 14:31:20 UTC 
Description of problem:
disk.img size is smaller than requested on 'px-csi-db-shared' storage class with {RWX, FS} when creating a 13Gi DV/PVC.

Version-Release number of selected component (if applicable):
4.14

How reproducible:
Always

Steps to Reproduce:

1. Create a DV with 13Gi storage request:

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: cnv-6536
  namespace: cdi-import-test-import-htt
spec:
  contentType: kubevirt
  source:
    http:
      certConfigMap: internal-https-configmap
      url: https://internal-http.cnv-tests-utilities/cirros-qcow2.img
  storage:
    resources:
      requests:
        storage: 13Gi
    storageClassName: px-csi-db-shared

2. Create a pod that uses this PVC (attaching the yaml)

3. Check the disk.img virtual size, see it's 12G instead of 13Gi:

$ oc exec cnv-6536-pod --namespace=cdi-import-test-import-htt -- qemu-img info /pvc/disk.img
image: /pvc/disk.img
file format: raw
virtual size: 12G (13153337344 bytes)
disk size: 13M

4. See the reported PVC capacity is 14Gi:

$ oc get pvc -n cdi-import-test-import-htt
NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
cnv-6536   Bound    pvc-292dbf65-8a7b-4e11-a27e-f0268858d160   14Gi       RWX            px-csi-db-shared   46m

$ oc get pvc -n cdi-import-test-import-htt cnv-6536 -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    cdi.kubevirt.io/storage.condition.running: "false"
    cdi.kubevirt.io/storage.condition.running.message: Import Complete
    cdi.kubevirt.io/storage.condition.running.reason: Completed
    cdi.kubevirt.io/storage.contentType: kubevirt
    cdi.kubevirt.io/storage.pod.phase: Succeeded
    cdi.kubevirt.io/storage.pod.restarts: "1"
    cdi.kubevirt.io/storage.populator.progress: 100.0%
    cdi.kubevirt.io/storage.preallocation.requested: "false"
    cdi.kubevirt.io/storage.usePopulator: "true"
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: pxd.portworx.com
    volume.kubernetes.io/storage-provisioner: pxd.portworx.com
  creationTimestamp: "2023-09-04T13:19:23Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    alerts.k8s.io/KubePersistentVolumeFillingUp: disabled
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-controller
    app.kubernetes.io/part-of: hyperconverged-cluster
    app.kubernetes.io/version: 4.14.0
    created-by-dynamic-class-creator: "Yes"
  name: cnv-6536
  namespace: cdi-import-test-import-htt
  ownerReferences:
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: DataVolume
    name: cnv-6536
    uid: cb1f6397-bd7b-45de-bd42-86185ceebfd4
  resourceVersion: "340909"
  uid: 824bb4e9-1132-4be8-a282-65b7e3acb31d
spec:
  accessModes:
  - ReadWriteMany
  dataSource:
    apiGroup: cdi.kubevirt.io
    kind: VolumeImportSource
    name: volume-import-source-cb1f6397-bd7b-45de-bd42-86185ceebfd4
  dataSourceRef:
    apiGroup: cdi.kubevirt.io
    kind: VolumeImportSource
    name: volume-import-source-cb1f6397-bd7b-45de-bd42-86185ceebfd4
  resources:
    requests:
      storage: "14771051548"
  storageClassName: px-csi-db-shared
  volumeMode: Filesystem
  volumeName: pvc-292dbf65-8a7b-4e11-a27e-f0268858d160
status:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 14Gi
  phase: Bound


Actual results:
Virtual image size is smaller than requested

Expected results:
Virtual image size is the same as requested

Additional info:
We don't see this issue with the requested size 64Mi and 1Gi - virtual image size is the same as requested.
Comment 3 Jenia Peimer 2023-09-26 10:24:32 UTC 
Also adding a PV yaml, it shows 14Gi:

$ oc get pv pvc-d3ed9759-e01c-4519-ae27-0a1abfdbc43c -oyaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: pxd.portworx.com
    volume.kubernetes.io/provisioner-deletion-secret-name: ""
    volume.kubernetes.io/provisioner-deletion-secret-namespace: ""
  creationTimestamp: "2023-09-26T10:20:38Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-d3ed9759-e01c-4519-ae27-0a1abfdbc43c
  resourceVersion: "96048"
  uid: fc717663-5456-474e-9b71-9597b28976c7
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 14Gi
  claimRef:
    name: cnv-6536
    namespace: cdi-import-test-import-htt
    resourceVersion: "96009"
    uid: f0491faa-2fa0-45a6-bd4e-da937183d374
  csi:
    driver: pxd.portworx.com
    fsType: ext4
    volumeAttributes:
      attached: ATTACH_STATE_INTERNAL_SWITCH
      error: ""
      parent: ""
      readonly: "false"
      secure: "false"
      shared: "false"
      sharedv4: "true"
      state: VOLUME_STATE_DETACHED
      storage.kubernetes.io/csiProvisionerIdentity: 1695722240804-8081-pxd.portworx.com
    volumeHandle: "79631756017134651"
  persistentVolumeReclaimPolicy: Delete
  storageClassName: px-csi-db-shared
  volumeMode: Filesystem
status:
  phase: Bound
Comment 4 dalia 2023-09-27 12:25:52 UTC 
Jenia, can you share the importer logs?
Comment 5 Jenia Peimer 2023-09-27 13:26:38 UTC 
I can only get the importer logs for a non-populators flow, but the virtual size is still 12Gi there:

$ oc exec dv-portworx-pod --namespace=cdi-import-test-import-htt -- qemu-img info /pvc/disk.img
image: /pvc/disk.img
file format: raw
virtual size: 12G (13153337344 bytes)
disk size: 13M


$ oc logs -n cdi-import-test-import-htt importer-dv-portworx
I0927 13:19:33.650455       1 importer.go:103] Starting importer
I0927 13:19:33.650487       1 importer.go:172] begin import process
I0927 13:19:33.663976       1 http-datasource.go:243] Attempting to get certs from /certs/tlsregistry.crt
I0927 13:19:33.683375       1 data-processor.go:356] Calculating available size
I0927 13:19:33.683400       1 data-processor.go:368] Checking out file system volume size.
I0927 13:19:33.683418       1 data-processor.go:376] Request image size not empty.
I0927 13:19:33.683426       1 data-processor.go:381] Target size 13919010816.
I0927 13:19:33.683445       1 data-processor.go:255] New phase: TransferScratch
I0927 13:19:33.683543       1 util.go:194] Writing data...
I0927 13:19:33.731084       1 data-processor.go:255] New phase: Convert
I0927 13:19:33.731103       1 data-processor.go:261] Validating image
E0927 13:19:33.882524       1 prlimit.go:155] failed to kill the process; os: process already finished
I0927 13:19:33.882698       1 qemu.go:126] Running qemu-img with args: [convert -t writeback -p -O raw /scratch/tmpimage /data/disk.img]
I0927 13:19:34.061356       1 qemu.go:258] 0.00
I0927 13:19:34.065650       1 qemu.go:258] 1.19
I0927 13:19:34.065661       1 qemu.go:258] 2.38
I0927 13:19:34.067208       1 qemu.go:258] 3.57
I0927 13:19:34.067456       1 qemu.go:258] 4.76
I0927 13:19:34.068156       1 qemu.go:258] 5.95
I0927 13:19:34.068614       1 qemu.go:258] 7.14
I0927 13:19:34.068874       1 qemu.go:258] 8.33
I0927 13:19:34.068967       1 qemu.go:258] 9.52
I0927 13:19:34.069107       1 qemu.go:258] 10.71
I0927 13:19:34.069415       1 qemu.go:258] 11.90
I0927 13:19:34.069523       1 qemu.go:258] 13.10
I0927 13:19:34.069672       1 qemu.go:258] 14.29
I0927 13:19:34.069734       1 qemu.go:258] 16.67
I0927 13:19:34.069916       1 qemu.go:258] 17.86
I0927 13:19:34.070117       1 qemu.go:258] 19.05
I0927 13:19:34.070530       1 qemu.go:258] 20.24
I0927 13:19:34.070782       1 qemu.go:258] 21.43
I0927 13:19:34.070969       1 qemu.go:258] 22.62
I0927 13:19:34.071171       1 qemu.go:258] 23.81
I0927 13:19:34.071517       1 qemu.go:258] 26.19
I0927 13:19:34.071747       1 qemu.go:258] 27.38
I0927 13:19:34.072015       1 qemu.go:258] 28.57
I0927 13:19:34.072630       1 qemu.go:258] 29.76
I0927 13:19:34.072681       1 qemu.go:258] 30.95
I0927 13:19:34.073051       1 qemu.go:258] 32.94
I0927 13:19:34.073121       1 qemu.go:258] 36.51
I0927 13:19:34.073348       1 qemu.go:258] 37.70
I0927 13:19:34.073453       1 qemu.go:258] 38.89
I0927 13:19:34.073849       1 qemu.go:258] 40.08
I0927 13:19:34.074200       1 qemu.go:258] 41.27
I0927 13:19:34.074448       1 qemu.go:258] 42.46
I0927 13:19:34.074627       1 qemu.go:258] 43.65
I0927 13:19:34.074727       1 qemu.go:258] 44.84
I0927 13:19:34.075019       1 qemu.go:258] 46.03
I0927 13:19:34.075151       1 qemu.go:258] 47.22
I0927 13:19:34.075166       1 qemu.go:258] 53.17
I0927 13:19:34.075196       1 qemu.go:258] 66.27
I0927 13:19:34.076627       1 qemu.go:258] 67.46
I0927 13:19:34.077970       1 qemu.go:258] 68.65
I0927 13:19:34.078085       1 qemu.go:258] 70.24
I0927 13:19:34.078204       1 qemu.go:258] 71.43
I0927 13:19:34.078632       1 qemu.go:258] 74.21
I0927 13:19:34.078863       1 qemu.go:258] 75.40
I0927 13:19:34.078965       1 qemu.go:258] 76.59
I0927 13:19:34.079306       1 qemu.go:258] 80.56
I0927 13:19:34.079481       1 qemu.go:258] 81.75
I0927 13:19:34.079680       1 qemu.go:258] 82.94
I0927 13:19:34.080313       1 qemu.go:258] 84.92
I0927 13:19:34.080673       1 qemu.go:258] 86.11
I0927 13:19:34.080934       1 qemu.go:258] 87.30
I0927 13:19:34.081567       1 qemu.go:258] 88.49
I0927 13:19:34.141309       1 qemu.go:258] 89.68
I0927 13:19:34.141536       1 qemu.go:258] 90.87
I0927 13:19:34.141735       1 qemu.go:258] 92.46
I0927 13:19:34.141819       1 qemu.go:258] 94.05
I0927 13:19:34.142149       1 qemu.go:258] 95.63
I0927 13:19:34.142543       1 qemu.go:258] 96.83
I0927 13:19:34.142803       1 qemu.go:258] 98.02
I0927 13:19:34.143046       1 qemu.go:258] 99.21
E0927 13:19:34.169387       1 prlimit.go:155] failed to kill the process; os: process already finished
I0927 13:19:34.169408       1 data-processor.go:255] New phase: Resize
E0927 13:19:34.341373       1 prlimit.go:155] failed to kill the process; os: process already finished
W0927 13:19:34.341422       1 data-processor.go:338] Available space less than requested size, resizing image to available space 13153337344.
I0927 13:19:34.341428       1 data-processor.go:349] Expanding image size to: 13153337344
E0927 13:19:34.498279       1 prlimit.go:155] failed to kill the process; os: process already finished
I0927 13:19:34.498295       1 data-processor.go:261] Validating image
E0927 13:19:34.673329       1 prlimit.go:155] failed to kill the process; os: process already finished
I0927 13:19:34.673455       1 data-processor.go:255] New phase: Complete
I0927 13:19:34.673579       1 importer.go:216] Import Complete
Comment 6 Adam Litke 2023-10-17 13:35:05 UTC 
Alex, can you work with Jenia to investigate this please.
Comment 7 Alex Kalenyuk 2023-10-18 15:12:52 UTC 
(In reply to Adam Litke from comment #6)
> Alex, can you work with Jenia to investigate this please.

Sure

So it sounds to me like Portworx does its own root reservation,
Combining that with how we derive "available space" on a Filesystem PVC (int64(stat.Bavail) * int64(stat.Bsize), note - Bavail instead of Bfree)
results in us resizing to a smaller target.

This has come up before:
https://github.com/kubevirt/containerized-data-importer/pull/2194
https://bugzilla.redhat.com/show_bug.cgi?id=2059057

@jpeimer could you help get a cluster with Portworx to ensure this is the case?
Comment 8 Jenia Peimer 2023-10-19 13:53:20 UTC 
Shared the details with Alex (in private) of how to get the cluster.
Comment 9 Pan Ousley 2023-10-30 19:25:18 UTC 
Hi Jenia/Alex, can you please provide doc text to help me write the known issue for 4.14? 

I set the doc type field to Known Issue, which has the following doc text fields:

> Cause: 

> Consequence: 

> Workaround (if any): 

> Result:


Thanks!
Comment 12 Adam Litke 2023-11-28 20:13:11 UTC 
Alex, besides this known issue, what are the follow-up actions that we should be taking to resolve this bug?
Comment 13 Alex Kalenyuk 2023-11-29 10:26:31 UTC 
(In reply to Adam Litke from comment #12)
> Alex, besides this known issue, what are the follow-up actions that we
> should be taking to resolve this bug?

When this came up before, we made the decision to *not* be aware of Bfree:
https://github.com/kubevirt/containerized-data-importer/pull/2194#issuecomment-1073958790
And this seems like a good direction to me. I think coordinating two different overheads is a recipe for trouble
Note You need to log in before you can comment on or make changes to this bug.