Bug 2237287
| Summary: | portworx: disk.img size is smaller than requested | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Jenia Peimer <jpeimer> |
| Component: | Storage | Assignee: | Alex Kalenyuk <akalenyu> |
| Status: | CLOSED MIGRATED | QA Contact: | Jenia Peimer <jpeimer> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.14.0 | CC: | akalenyu, alitke, dafrank, pousley, ycui |
| Target Milestone: | --- | Flags: | pousley:
needinfo-
pousley: needinfo- |
| Target Release: | 4.14.2 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Known Issue | |
| Doc Text: |
Cause: Due to double accounting of filesystem overhead with Portworx storage
Consequence: in some cases a VM disk image may be created with a size smaller than intended.
Workaround (if any): The PVC may be expanded to add additional space after initial provisioning is completed.
Result: Extra space can be made available to correct the issue.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-12-14 16:11:37 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Also adding a PV yaml, it shows 14Gi:
$ oc get pv pvc-d3ed9759-e01c-4519-ae27-0a1abfdbc43c -oyaml
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: pxd.portworx.com
volume.kubernetes.io/provisioner-deletion-secret-name: ""
volume.kubernetes.io/provisioner-deletion-secret-namespace: ""
creationTimestamp: "2023-09-26T10:20:38Z"
finalizers:
- kubernetes.io/pv-protection
name: pvc-d3ed9759-e01c-4519-ae27-0a1abfdbc43c
resourceVersion: "96048"
uid: fc717663-5456-474e-9b71-9597b28976c7
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 14Gi
claimRef:
name: cnv-6536
namespace: cdi-import-test-import-htt
resourceVersion: "96009"
uid: f0491faa-2fa0-45a6-bd4e-da937183d374
csi:
driver: pxd.portworx.com
fsType: ext4
volumeAttributes:
attached: ATTACH_STATE_INTERNAL_SWITCH
error: ""
parent: ""
readonly: "false"
secure: "false"
shared: "false"
sharedv4: "true"
state: VOLUME_STATE_DETACHED
storage.kubernetes.io/csiProvisionerIdentity: 1695722240804-8081-pxd.portworx.com
volumeHandle: "79631756017134651"
persistentVolumeReclaimPolicy: Delete
storageClassName: px-csi-db-shared
volumeMode: Filesystem
status:
phase: Bound
Jenia, can you share the importer logs? I can only get the importer logs for a non-populators flow, but the virtual size is still 12Gi there: $ oc exec dv-portworx-pod --namespace=cdi-import-test-import-htt -- qemu-img info /pvc/disk.img image: /pvc/disk.img file format: raw virtual size: 12G (13153337344 bytes) disk size: 13M $ oc logs -n cdi-import-test-import-htt importer-dv-portworx I0927 13:19:33.650455 1 importer.go:103] Starting importer I0927 13:19:33.650487 1 importer.go:172] begin import process I0927 13:19:33.663976 1 http-datasource.go:243] Attempting to get certs from /certs/tlsregistry.crt I0927 13:19:33.683375 1 data-processor.go:356] Calculating available size I0927 13:19:33.683400 1 data-processor.go:368] Checking out file system volume size. I0927 13:19:33.683418 1 data-processor.go:376] Request image size not empty. I0927 13:19:33.683426 1 data-processor.go:381] Target size 13919010816. I0927 13:19:33.683445 1 data-processor.go:255] New phase: TransferScratch I0927 13:19:33.683543 1 util.go:194] Writing data... I0927 13:19:33.731084 1 data-processor.go:255] New phase: Convert I0927 13:19:33.731103 1 data-processor.go:261] Validating image E0927 13:19:33.882524 1 prlimit.go:155] failed to kill the process; os: process already finished I0927 13:19:33.882698 1 qemu.go:126] Running qemu-img with args: [convert -t writeback -p -O raw /scratch/tmpimage /data/disk.img] I0927 13:19:34.061356 1 qemu.go:258] 0.00 I0927 13:19:34.065650 1 qemu.go:258] 1.19 I0927 13:19:34.065661 1 qemu.go:258] 2.38 I0927 13:19:34.067208 1 qemu.go:258] 3.57 I0927 13:19:34.067456 1 qemu.go:258] 4.76 I0927 13:19:34.068156 1 qemu.go:258] 5.95 I0927 13:19:34.068614 1 qemu.go:258] 7.14 I0927 13:19:34.068874 1 qemu.go:258] 8.33 I0927 13:19:34.068967 1 qemu.go:258] 9.52 I0927 13:19:34.069107 1 qemu.go:258] 10.71 I0927 13:19:34.069415 1 qemu.go:258] 11.90 I0927 13:19:34.069523 1 qemu.go:258] 13.10 I0927 13:19:34.069672 1 qemu.go:258] 14.29 I0927 13:19:34.069734 1 qemu.go:258] 16.67 I0927 13:19:34.069916 1 qemu.go:258] 17.86 I0927 13:19:34.070117 1 qemu.go:258] 19.05 I0927 13:19:34.070530 1 qemu.go:258] 20.24 I0927 13:19:34.070782 1 qemu.go:258] 21.43 I0927 13:19:34.070969 1 qemu.go:258] 22.62 I0927 13:19:34.071171 1 qemu.go:258] 23.81 I0927 13:19:34.071517 1 qemu.go:258] 26.19 I0927 13:19:34.071747 1 qemu.go:258] 27.38 I0927 13:19:34.072015 1 qemu.go:258] 28.57 I0927 13:19:34.072630 1 qemu.go:258] 29.76 I0927 13:19:34.072681 1 qemu.go:258] 30.95 I0927 13:19:34.073051 1 qemu.go:258] 32.94 I0927 13:19:34.073121 1 qemu.go:258] 36.51 I0927 13:19:34.073348 1 qemu.go:258] 37.70 I0927 13:19:34.073453 1 qemu.go:258] 38.89 I0927 13:19:34.073849 1 qemu.go:258] 40.08 I0927 13:19:34.074200 1 qemu.go:258] 41.27 I0927 13:19:34.074448 1 qemu.go:258] 42.46 I0927 13:19:34.074627 1 qemu.go:258] 43.65 I0927 13:19:34.074727 1 qemu.go:258] 44.84 I0927 13:19:34.075019 1 qemu.go:258] 46.03 I0927 13:19:34.075151 1 qemu.go:258] 47.22 I0927 13:19:34.075166 1 qemu.go:258] 53.17 I0927 13:19:34.075196 1 qemu.go:258] 66.27 I0927 13:19:34.076627 1 qemu.go:258] 67.46 I0927 13:19:34.077970 1 qemu.go:258] 68.65 I0927 13:19:34.078085 1 qemu.go:258] 70.24 I0927 13:19:34.078204 1 qemu.go:258] 71.43 I0927 13:19:34.078632 1 qemu.go:258] 74.21 I0927 13:19:34.078863 1 qemu.go:258] 75.40 I0927 13:19:34.078965 1 qemu.go:258] 76.59 I0927 13:19:34.079306 1 qemu.go:258] 80.56 I0927 13:19:34.079481 1 qemu.go:258] 81.75 I0927 13:19:34.079680 1 qemu.go:258] 82.94 I0927 13:19:34.080313 1 qemu.go:258] 84.92 I0927 13:19:34.080673 1 qemu.go:258] 86.11 I0927 13:19:34.080934 1 qemu.go:258] 87.30 I0927 13:19:34.081567 1 qemu.go:258] 88.49 I0927 13:19:34.141309 1 qemu.go:258] 89.68 I0927 13:19:34.141536 1 qemu.go:258] 90.87 I0927 13:19:34.141735 1 qemu.go:258] 92.46 I0927 13:19:34.141819 1 qemu.go:258] 94.05 I0927 13:19:34.142149 1 qemu.go:258] 95.63 I0927 13:19:34.142543 1 qemu.go:258] 96.83 I0927 13:19:34.142803 1 qemu.go:258] 98.02 I0927 13:19:34.143046 1 qemu.go:258] 99.21 E0927 13:19:34.169387 1 prlimit.go:155] failed to kill the process; os: process already finished I0927 13:19:34.169408 1 data-processor.go:255] New phase: Resize E0927 13:19:34.341373 1 prlimit.go:155] failed to kill the process; os: process already finished W0927 13:19:34.341422 1 data-processor.go:338] Available space less than requested size, resizing image to available space 13153337344. I0927 13:19:34.341428 1 data-processor.go:349] Expanding image size to: 13153337344 E0927 13:19:34.498279 1 prlimit.go:155] failed to kill the process; os: process already finished I0927 13:19:34.498295 1 data-processor.go:261] Validating image E0927 13:19:34.673329 1 prlimit.go:155] failed to kill the process; os: process already finished I0927 13:19:34.673455 1 data-processor.go:255] New phase: Complete I0927 13:19:34.673579 1 importer.go:216] Import Complete Alex, can you work with Jenia to investigate this please. (In reply to Adam Litke from comment #6) > Alex, can you work with Jenia to investigate this please. Sure So it sounds to me like Portworx does its own root reservation, Combining that with how we derive "available space" on a Filesystem PVC (int64(stat.Bavail) * int64(stat.Bsize), note - Bavail instead of Bfree) results in us resizing to a smaller target. This has come up before: https://github.com/kubevirt/containerized-data-importer/pull/2194 https://bugzilla.redhat.com/show_bug.cgi?id=2059057 @jpeimer could you help get a cluster with Portworx to ensure this is the case? Shared the details with Alex (in private) of how to get the cluster. Hi Jenia/Alex, can you please provide doc text to help me write the known issue for 4.14? I set the doc type field to Known Issue, which has the following doc text fields: > Cause: > Consequence: > Workaround (if any): > Result: Thanks! Alex, besides this known issue, what are the follow-up actions that we should be taking to resolve this bug? (In reply to Adam Litke from comment #12) > Alex, besides this known issue, what are the follow-up actions that we > should be taking to resolve this bug? When this came up before, we made the decision to *not* be aware of Bfree: https://github.com/kubevirt/containerized-data-importer/pull/2194#issuecomment-1073958790 And this seems like a good direction to me. I think coordinating two different overheads is a recipe for trouble |
Description of problem: disk.img size is smaller than requested on 'px-csi-db-shared' storage class with {RWX, FS} when creating a 13Gi DV/PVC. Version-Release number of selected component (if applicable): 4.14 How reproducible: Always Steps to Reproduce: 1. Create a DV with 13Gi storage request: apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: cnv-6536 namespace: cdi-import-test-import-htt spec: contentType: kubevirt source: http: certConfigMap: internal-https-configmap url: https://internal-http.cnv-tests-utilities/cirros-qcow2.img storage: resources: requests: storage: 13Gi storageClassName: px-csi-db-shared 2. Create a pod that uses this PVC (attaching the yaml) 3. Check the disk.img virtual size, see it's 12G instead of 13Gi: $ oc exec cnv-6536-pod --namespace=cdi-import-test-import-htt -- qemu-img info /pvc/disk.img image: /pvc/disk.img file format: raw virtual size: 12G (13153337344 bytes) disk size: 13M 4. See the reported PVC capacity is 14Gi: $ oc get pvc -n cdi-import-test-import-htt NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE cnv-6536 Bound pvc-292dbf65-8a7b-4e11-a27e-f0268858d160 14Gi RWX px-csi-db-shared 46m $ oc get pvc -n cdi-import-test-import-htt cnv-6536 -oyaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: cdi.kubevirt.io/storage.condition.running: "false" cdi.kubevirt.io/storage.condition.running.message: Import Complete cdi.kubevirt.io/storage.condition.running.reason: Completed cdi.kubevirt.io/storage.contentType: kubevirt cdi.kubevirt.io/storage.pod.phase: Succeeded cdi.kubevirt.io/storage.pod.restarts: "1" cdi.kubevirt.io/storage.populator.progress: 100.0% cdi.kubevirt.io/storage.preallocation.requested: "false" cdi.kubevirt.io/storage.usePopulator: "true" pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: pxd.portworx.com volume.kubernetes.io/storage-provisioner: pxd.portworx.com creationTimestamp: "2023-09-04T13:19:23Z" finalizers: - kubernetes.io/pvc-protection labels: alerts.k8s.io/KubePersistentVolumeFillingUp: disabled app: containerized-data-importer app.kubernetes.io/component: storage app.kubernetes.io/managed-by: cdi-controller app.kubernetes.io/part-of: hyperconverged-cluster app.kubernetes.io/version: 4.14.0 created-by-dynamic-class-creator: "Yes" name: cnv-6536 namespace: cdi-import-test-import-htt ownerReferences: - apiVersion: cdi.kubevirt.io/v1beta1 blockOwnerDeletion: true controller: true kind: DataVolume name: cnv-6536 uid: cb1f6397-bd7b-45de-bd42-86185ceebfd4 resourceVersion: "340909" uid: 824bb4e9-1132-4be8-a282-65b7e3acb31d spec: accessModes: - ReadWriteMany dataSource: apiGroup: cdi.kubevirt.io kind: VolumeImportSource name: volume-import-source-cb1f6397-bd7b-45de-bd42-86185ceebfd4 dataSourceRef: apiGroup: cdi.kubevirt.io kind: VolumeImportSource name: volume-import-source-cb1f6397-bd7b-45de-bd42-86185ceebfd4 resources: requests: storage: "14771051548" storageClassName: px-csi-db-shared volumeMode: Filesystem volumeName: pvc-292dbf65-8a7b-4e11-a27e-f0268858d160 status: accessModes: - ReadWriteMany capacity: storage: 14Gi phase: Bound Actual results: Virtual image size is smaller than requested Expected results: Virtual image size is the same as requested Additional info: We don't see this issue with the requested size 64Mi and 1Gi - virtual image size is the same as requested.