Bug 1891520 - Moving or copying raw preallocated disk to file storage make the disk sparse
Summary: Moving or copying raw preallocated disk to file storage make the disk sparse
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: vdsm
Classification: oVirt
Component: Core
Version: 4.40.35
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ovirt-4.4.3-1
: ---
Assignee: Nir Soffer
QA Contact: Evelina Shames
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-26 14:06 UTC by Nir Soffer
Modified: 2020-11-11 06:42 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-11 06:42:05 UTC
oVirt Team: Storage
Embargoed:
pm-rhel: ovirt-4.4+
aoconnor: blocker+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 111897 0 master MERGED tests: Mark failing copy_data tests 2020-12-08 13:09:58 UTC
oVirt gerrit 111903 0 master MERGED qemuimg: Fix copy to raw preallocated volume 2020-12-08 13:10:27 UTC
oVirt gerrit 111943 0 ovirt-4.4.3 MERGED tests: Mark failing copy_data tests 2020-12-08 13:10:27 UTC
oVirt gerrit 111944 0 ovirt-4.4.3 MERGED qemuimg: Fix copy to raw preallocated volume 2020-12-08 13:09:58 UTC

Description Nir Soffer 2020-10-26 14:06:50 UTC 
Description of problem:

Moving or copying a raw preallocated disk to file storage creates
a preallocated volume on the destination storage, but after the copy
the actual volume is sparse.

Version-Release number of selected component (if applicable):
4.40.35

How reproducible:
Always

Steps to Reproduce:
1. Create new disk on block storage
2. Move the disk to file storage (tested with NFS v4.2)

Actual results:

The disk is preallocaed:
$ egrep ^TYPE= /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/df007b8d-c1d8-47e6-9f09-c5ec75f07af3/046bbd89-4672-44f5-b757-8e57a67517a0.meta
TYPE=PREALLOCATED

But the file is sparse:

$ ls -lhs /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/df007b8d-c1d8-47e6-9f09-c5ec75f07af3/046bbd89-4672-44f5-b757-8e57a67517a0
4.0K -rw-rw----. 1 vdsm kvm 2.0G Oct 26 15:29 /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/df007b8d-c1d8-47e6-9f09-c5ec75f07af3/046bbd89-4672-44f5-b757-8e57a67517a0

Expected results:

The file must be fully allocated - the file size must be equal to the
allocated size (sometimes it may be even little bigger).

Additional info:

Since this change:

commit 84ef2dd05fe1f4dc9e9158d91a7ec84f3c9a4650
Author: Nir Soffer <nsoffer>
Date:   Sun Jun 21 13:36:58 2020 +0300

    fileSD: Do not use qemu-img preallocation=falloc

We don't use qemu-img -o preallocation=falloc to create the destination
image during copy or move disk. Instead we create the image preallocated,
and use the -n option to convert to exiting image.

When this change as done, copying preallocated images worked
(tested with qemu-img 4.2.0-19). Our CI in travis validated this
use case with this qemu-img version. When we updated the the CI container
image to qemu-img 4.2.0-29, the test for this case started to fail.

When using "qemu-img convert -n", qemu-img does not know the contents
of the image, so it must zero the target image. Zeroing is done by punching
holes, so the image becomes sparse.

I'm not sure why this worked with qemu-img 4.2.0-19, it may be related to 
this performance improvement:

commit edafc70c0c8510862f2f213a3acf7067113bcd08
Author: Kevin Wolf <kwolf>
Date:   Mon Jun 22 17:12:03 2020 +0200

    qemu-img convert: Don't pre-zero images

However qemu-img 5.1 (available in RHEL AV 8.3) introduced the
--target-is-zeo option. So we can fix this issue using:

   qemu-img convert -n --target-is-zero ...

When coping disks to file storage.

I think oVirt/RHV 4.4.2 and oVirt 4.4.3 rc versions are affected.
Comment 1 Eyal Shenitzky 2020-10-27 12:08:13 UTC 
We should verify all the SDM flows that use 'qemu-img covert'.
And verify that the size matches what we expect.

For example - 

1. Move disks while the VM is down (cold move)
2. Create VM from a template (thin/clone)
Comment 2 Nir Soffer 2020-10-27 15:48:55 UTC 
Here is I tested the fix:

## Preallocated VM with snapshot on iscsi storage

Check that preallocated disks remain preallocated when copied to new disk.


# qemu-img info -U --backing-chain /dev/27f2b637-ffb1-48f9-8f68-63ed227392b9/04f15e1d-670c-4304-a74d-0ec20686dcdc
image: /dev/27f2b637-ffb1-48f9-8f68-63ed227392b9/04f15e1d-670c-4304-a74d-0ec20686dcdc
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 0 B
cluster_size: 65536
backing file: 7468e5ff-60b1-47f5-86af-8e38ab51e2f5 (actual path: /dev/27f2b637-ffb1-48f9-8f68-63ed227392b9/7468e5ff-60b1-47f5-86af-8e38ab51e2f5)
backing file format: raw
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

image: /dev/27f2b637-ffb1-48f9-8f68-63ed227392b9/7468e5ff-60b1-47f5-86af-8e38ab51e2f5
file format: raw
virtual size: 6 GiB (6442450944 bytes)
disk size: 0 B

# lvs -o lv_name,tags,size | grep 7ce84142-2d34-4fba-b46e-d331419e0675
  04f15e1d-670c-4304-a74d-0ec20686dcdc IU_7ce84142-2d34-4fba-b46e-d331419e0675,MD_10,PU_7468e5ff-60b1-47f5-86af-8e38ab51e2f5   2.12g
  7468e5ff-60b1-47f5-86af-8e38ab51e2f5 IU_7ce84142-2d34-4fba-b46e-d331419e0675,MD_9,PU_00000000-0000-0000-0000-000000000000    6.00g


After moving disk to NFS storage (live):

# qemu-img info -U --backing-chain /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/7ce84142-2d34-4fba-b46e-d331419e0675/04f15e1d-670c-4304-a74d-0ec20686dcdc
image: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/7ce84142-2d34-4fba-b46e-d331419e0675/04f15e1d-670c-4304-a74d-0ec20686dcdc
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 104 MiB
cluster_size: 65536
backing file: 7468e5ff-60b1-47f5-86af-8e38ab51e2f5 (actual path: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/7ce84142-2d34-4fba-b46e-d331419e0675/7468e5ff-60b1-47f5-86af-8e38ab51e2f5)
backing file format: raw
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

image: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/7ce84142-2d34-4fba-b46e-d331419e0675/7468e5ff-60b1-47f5-86af-8e38ab51e2f5
file format: raw
virtual size: 6 GiB (6442450944 bytes)
disk size: 6 GiB


After moving to NFS (cold):

# qemu-img info -U --backing-chain /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/7ce84142-2d34-4fba-b46e-d331419e0675/04f15e1d-670c-4304-a74d-0ec20686dcdc
image: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/7ce84142-2d34-4fba-b46e-d331419e0675/04f15e1d-670c-4304-a74d-0ec20686dcdc
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 40.4 MiB
cluster_size: 65536
backing file: 7468e5ff-60b1-47f5-86af-8e38ab51e2f5 (actual path: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/7ce84142-2d34-4fba-b46e-d331419e0675/7468e5ff-60b1-47f5-86af-8e38ab51e2f5)
backing file format: raw
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

image: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/7ce84142-2d34-4fba-b46e-d331419e0675/7468e5ff-60b1-47f5-86af-8e38ab51e2f5
file format: raw
virtual size: 6 GiB (6442450944 bytes)
disk size: 6 GiB


(Deleting snapshot, converting disk to preallocated)


Creating template from VM (raw):

# qemu-img info /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/2130c252-dd6b-4e60-aa0b-2d0eac931b3d/a3b5c10a-1776-4bc1-9e03-209232b6ebd1
image: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/2130c252-dd6b-4e60-aa0b-2d0eac931b3d/a3b5c10a-1776-4bc1-9e03-209232b6ebd1
file format: raw
virtual size: 6 GiB (6442450944 bytes)
disk size: 6 GiB

# egrep ^TYPE= /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/2130c252-dd6b-4e60-aa0b-2d0eac931b3d/a3b5c10a-1776-4bc1-9e03-209232b6ebd1.meta
TYPE=PREALLOCATED


Creating template from VM (qcow2):

# qemu-img info /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/e9805fd3-4acc-4197-8ef0-0dcc3fbb2631/4a582b73-6dd0-4acd-8156-62a9dbae399c
image: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/e9805fd3-4acc-4197-8ef0-0dcc3fbb2631/4a582b73-6dd0-4acd-8156-62a9dbae399c
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 1.86 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

# egrep ^TYPE= /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/e9805fd3-4acc-4197-8ef0-0dcc3fbb2631/4a582b73-6dd0-4acd-8156-62a9dbae399c.meta
TYPE=SPARSE


Creating VM from template (raw):

# qemu-img info /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/af2b0362-97a2-451b-a85d-3947c17e39be/f2ecc57b-3500-4915-8fc8-d644e934e313
image: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/af2b0362-97a2-451b-a85d-3947c17e39be/f2ecc57b-3500-4915-8fc8-d644e934e313
file format: raw
virtual size: 6 GiB (6442450944 bytes)
disk size: 6 GiB

# egrep ^TYPE= /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/af2b0362-97a2-451b-a85d-3947c17e39be/f2ecc57b-3500-4915-8fc8-d644e934e313.meta
TYPE=PREALLOCATED


Creating VM from template (qcow2):

# qemu-img info /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/0428665e-ba83-4023-af9b-009c689e66e8/925a3162-b4c6-4bdd-9d88-3f304da0394c
image: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/0428665e-ba83-4023-af9b-009c689e66e8/925a3162-b4c6-4bdd-9d88-3f304da0394c
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 1.66 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

# egrep ^TYPE= /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/0428665e-ba83-4023-af9b-009c689e66e8/925a3162-b4c6-4bdd-9d88-3f304da0394c.meta
TYPE=SPARSE


## Sparse VM with snapshot on iscsi storage

Check regression when handling sparse disks.

# qemu-img info -U --backing-chain /dev/27f2b637-ffb1-48f9-8f68-63ed227392b9/0a44d697-41bd-4f38-81f2-e955f51799c4
image: /dev/27f2b637-ffb1-48f9-8f68-63ed227392b9/0a44d697-41bd-4f38-81f2-e955f51799c4
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 0 B
cluster_size: 65536
backing file: 20391fbc-fa77-4fef-9aea-cf59f27f90b5 (actual path: /dev/27f2b637-ffb1-48f9-8f68-63ed227392b9/20391fbc-fa77-4fef-9aea-cf59f27f90b5)
backing file format: qcow2
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

image: /dev/27f2b637-ffb1-48f9-8f68-63ed227392b9/20391fbc-fa77-4fef-9aea-cf59f27f90b5
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 0 B
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

# lvs -o lv_name,tags,size | grep 4b62aa6d-3bdd-4db3-b26f-0484c4124631
  0a44d697-41bd-4f38-81f2-e955f51799c4 IU_4b62aa6d-3bdd-4db3-b26f-0484c4124631,MD_5,PU_20391fbc-fa77-4fef-9aea-cf59f27f90b5    2.12g
  20391fbc-fa77-4fef-9aea-cf59f27f90b5 IU_4b62aa6d-3bdd-4db3-b26f-0484c4124631,MD_4,PU_00000000-0000-0000-0000-000000000000   <1.88g


After moving to NFS storage (live):

# qemu-img info -U --backing-chain /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/4b62aa6d-3bdd-4db3-b26f-0484c4124631/0a44d697-41bd-4f38-81f2-e955f51799c4
image: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/4b62aa6d-3bdd-4db3-b26f-0484c4124631/0a44d697-41bd-4f38-81f2-e955f51799c4
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 72.2 MiB
cluster_size: 65536
backing file: 20391fbc-fa77-4fef-9aea-cf59f27f90b5 (actual path: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/4b62aa6d-3bdd-4db3-b26f-0484c4124631/20391fbc-fa77-4fef-9aea-cf59f27f90b5)
backing file format: qcow2
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

image: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/4b62aa6d-3bdd-4db3-b26f-0484c4124631/20391fbc-fa77-4fef-9aea-cf59f27f90b5
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 1.65 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false


After moving to NFS storage (cold):

# qemu-img info -U --backing-chain /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/4b62aa6d-3bdd-4db3-b26f-0484c4124631/0a44d697-41bd-4f38-81f2-e955f51799c4
image: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/4b62aa6d-3bdd-4db3-b26f-0484c4124631/0a44d697-41bd-4f38-81f2-e955f51799c4
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 72.4 MiB
cluster_size: 65536
backing file: 20391fbc-fa77-4fef-9aea-cf59f27f90b5 (actual path: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/4b62aa6d-3bdd-4db3-b26f-0484c4124631/20391fbc-fa77-4fef-9aea-cf59f27f90b5)
backing file format: qcow2
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

image: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/4b62aa6d-3bdd-4db3-b26f-0484c4124631/20391fbc-fa77-4fef-9aea-cf59f27f90b5
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 1.65 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false


Creating template from VM (raw):

# qemu-img info /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/d6c4edb9-e66a-4bf8-ad53-1a707c060c7d/44c453ea-759a-42b6-a86f-d1a200f8c77d
image: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/d6c4edb9-e66a-4bf8-ad53-1a707c060c7d/44c453ea-759a-42b6-a86f-d1a200f8c77d
file format: raw
virtual size: 6 GiB (6442450944 bytes)
disk size: 1.63 GiB

# egrep ^TYPE= /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/d6c4edb9-e66a-4bf8-ad53-1a707c060c7d/44c453ea-759a-42b6-a86f-d1a200f8c77d.meta
TYPE=SPARSE


Creating template from VM (qcow2):

# qemu-img info /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/d2ab51e3-66d2-48bb-871a-982e9921597c/b480b5ac-7f17-4204-a1f2-3cd432ef6049
image: /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/d2ab51e3-66d2-48bb-871a-982e9921597c/b480b5ac-7f17-4204-a1f2-3cd432ef6049
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 1.65 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

# egrep ^TYPE= /rhev/data-center/mnt/nfs1:_export_3/f5915245-0ac5-4712-b8b2-dd4d4be7cdc4/images/d2ab51e3-66d2-48bb-871a-982e9921597c/b480b5ac-7f17-4204-a1f2-3cd432ef6049.meta
TYPE=SPARSE
Comment 3 Nir Soffer 2020-10-27 16:32:47 UTC 
Note: Fix is available only on RHEL 8.3, providing qemu-img 5.1.0.
We cannot fix this issue on CentOS 8.2.
Comment 4 Evelina Shames 2020-11-01 10:32:00 UTC 
Verified on rhv-4.4.3-12
Comment 5 Sandro Bonazzola 2020-11-11 06:42:05 UTC 
This bugzilla is included in oVirt 4.4.3 release, published on November 10th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.3 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.
Note You need to log in before you can comment on or make changes to this bug.