1433670 – Provide an API that estimates the size of QCOW2 image converted from a raw image

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1433670 - Provide an API that estimates the size of QCOW2 image converted from a raw image

Summary: Provide an API that estimates the size of QCOW2 image converted from a raw i...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	qemu-kvm-rhev
Sub Component:
Version:	7.4
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Stefan Hajnoczi
QA Contact:	Ping Li
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1358717 1419240 1475947 1475974 1547960
TreeView+	depends on / blocked

Reported:	2017-03-19 07:51 UTC by Maor
Modified:	2018-04-11 00:15 UTC (History)
CC List:	19 users (show)
Fixed In Version:	qemu-kvm-rhev-2.10.0-1.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1475947 1475974 (view as bug list)
Environment:
Last Closed:	2018-04-11 00:12:33 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	1547960	0	unspecified	CLOSED	Use qemu-img measure to estimate image size	2021-02-22 00:41:40 UTC
Red Hat Product Errata	RHSA-2018:1104	0	None	None	None	2018-04-11 00:15:02 UTC

Internal Links: 1547960

Description Maor 2017-03-19 07:51:47 UTC

Description of problem:
I was wondering if that will be possible to provide a new API that estimates the size of qcow2 image converted from a raw image.
We could use this new API to allocate the size more precisely before the convert operation.

The operation that we are trying to do is to convert a raw sparse image from NFS or from block device to qcow2 image on thin provisioned block device

In ovirt thin provisioned block device is a regular lv, and we like
to allocate only the required size for the the qcow file.

Our current (stupid) solution is to allocate the entire LV using the
size of the raw image.


Here is an example flow:

    $ truncate -s 10G test.raw

We don't know what will be the size of the qcow on the block storage,
so we allocate the entire LV:

    $ lvcreate --size 10G vg/lv

Then we convert the file to the new LV:

    $ qemu-img convert -f raw -O qcow2 test.raw /dev/vg/lv

After the copy we can check the actual size:

    $ qemu-img check /dev/vg/lv

And reduce the LV:

    $ lvreduce -L128m vg/lv

But we like to avoid the allocation, and allocate only the needed size
before we convert the image.

We think that the best way to solve this issue is to return this info
from qemu-img, maybe as a flag to qemu-img convert that will
calculate the size of the converted image without doing any writes.


Version-Release number of selected component (if applicable):

Comment 2 Stefan Hajnoczi 2017-07-14 09:01:00 UTC

The following commits are now in qemu.git/master and will be included in the next qemu-kvm-rhev rebase:

32a1681 iotests: add test 178 for qemu-img measure
217a068 qemu-iotests: support per-format golden output files
fd03c2b qemu-img: add measure subcommand
c501c35 qcow2: add bdrv_measure() support
0eb4a8c qcow2: extract image creation option parsing
7c5bcc4 qcow2: make refcount size calculation conservative
95c67e3 qcow2: extract preallocation calculation function
a843a22 raw-format: add bdrv_measure() support
90880ff block: add bdrv_measure() API

Comment 3 Stefan Hajnoczi 2017-09-20 14:54:13 UTC

Adding Yaniv to CC so he gets an email when this BZ moves from POST to MODIFIED.  At that point qemu-kvm-rhev 2.10 can be tested (it will include this fix).

Comment 5 Ping Li 2017-10-10 16:21:04 UTC

Hi Stefan,

According to the test result, file size calculated by "qemu-img measure" is greater than the actual value. Could you help to check whether it is expected?

Packages tested:
kernel-3.10.0-731.el7.x86_64
qemu-kvm-rhev-2.10.0-1.el7

Test steps:
1 Calculate the file size for a new empty image.
1.1 for raw format
# qemu-img measure --size 4G -O raw
required size: 4294967296
fully allocated size: 4294967296

# qemu-img create -f raw test.img 4G
Formatting 'test.img', fmt=raw size=4294967296
[root@localhost test]# qemu-img info test.img 
image: test.img
file format: raw
virtual size: 4.0G (4294967296 bytes)
disk size: 0
# qemu-img check test.img 
qemu-img: This image format does not support checks
# ll test.img 
-rw-r--r-- 1 root root 4294967296 Oct 10 11:13 test.img

1.2 for qcow2 format
# qemu-img measure --size 4G -O qcow2
required size: 917504  ----------> 917504 bytes
fully allocated size: 4295884800  ----------> 4295884800 bytes

# qemu-img create -f qcow2 test.qcow2 4G
Formatting 'test.qcow2', fmt=qcow2 size=4294967296 cluster_size=65536 lazy_refcounts=off refcount_bits=16
# qemu-img info test.qcow2 
image: test.qcow2
file format: qcow2
virtual size: 4.0G (4294967296 bytes)  ----------> 4294967296 bytes
disk size: 196K
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
# qemu-img check test.qcow2 
No errors were found on the image.
Image end offset: 262144----------> 262144 bytes
# ll test.qcow2                     
-rw-r--r-- 1 root root 196672 Oct 10 11:15 test.qcow2

2 Calculate the file size for a converted image.
2.1 Both original and new images are on local filesystem
2.1.1 convert qcow2 to qcow2
# qemu-img measure -f qcow2 -O qcow2 rhel75.qcow2 
required size: 1335033856  ----------> 1335033856 bytes
fully allocated size: 21478375424  ----------> 21478375424 bytes

# qemu-img convert -p -f qcow2 -O qcow2 rhel75.qcow2 new.qcow2
    (100.00/100%)
# qemu-img info new.qcow2 
image: new.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)  ----------> 21474836480 bytes
disk size: 2.0G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
# qemu-img check new.qcow2 
No errors were found on the image.
20085/327680 = 6.13% allocated, 0.00% fragmented, 0.00% compressed clusters
Image end offset: 1317011456  ----------> 1317011456 bytes
# ll new.qcow2 
-rw-r--r-- 1 root root 1317011456 Oct 10 11:39 new.qcow2

2.1.2 convert qcow2 to raw
# qemu-img measure -f qcow2 -O raw rhel75.qcow2 
required size: 21474836480
fully allocated size: 21474836480

# qemu-img convert -p -f qcow2 -O raw rhel75.qcow2 new.img
    (100.00/100%)
# qemu-img info new.img 
image: new.img
file format: raw
virtual size: 20G (21474836480 bytes)
disk size: 1.2G
# ll new.img 
-rw-r--r-- 1 root root 21474836480 Oct 10 11:43 new.img

2.1.3 convert raw to qcow2
# qemu-img measure -f raw -O qcow2 rhel74-64-virtio.raw 
required size: 6923812864  ----------> 6923812864 bytes
fully allocated size: 21478375424  ----------> 21478375424 bytes

# qemu-img convert -p -f raw -O qcow2 rhel74-64-virtio.raw new.qcow2
    (100.00/100%)
# qemu-img info new.qcow2 
image: new.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)  ----------> 21474836480 bytes
disk size: 8.0G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
# qemu-img check new.qcow2 
No errors were found on the image.
68526/327680 = 20.91% allocated, 0.00% fragmented, 0.00% compressed clusters
Image end offset: 4492230656  ----------> 4492230656 bytes
# ll new.qcow2 
-rw-r--r-- 1 root root 4492230656 Oct 10 11:53 new.qcow2

2.1.4 convert raw to raw
# qemu-img measure -f raw -O raw rhel74-64-virtio.raw 
required size: 21474836480
fully allocated size: 21474836480

# qemu-img convert -p -f raw -O raw rhel74-64-virtio.raw new.img 
    (100.00/100%)
# qemu-img info new.img 
image: new.img
file format: raw
virtual size: 20G (21474836480 bytes)
disk size: 4.2G
# ll new.img 
-rw-r--r-- 1 root root 21474836480 Oct 10 11:56 new.img

2.2 New image is on logical volumes
2.2.1 convert raw to qcow2
# qemu-img measure -f raw -O qcow2 test.raw 
required size: 917504  ----------> 917504 bytes
fully allocated size: 4295884800  ----------> 4295884800 bytes

# qemu-img convert -p -f raw -O qcow2 test.raw /dev/vg/lv 
    (100.00/100%)
# qemu-img info /dev/vg/lv 
image: /dev/vg/lv
file format: qcow2
virtual size: 4.0G (4294967296 bytes)  ----------> 4294967296 bytes
disk size: 0
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
# qemu-img check /dev/vg/lv 
No errors were found on the image.
Image end offset: 262144  ----------> 262144 bytes

2.2.2 convert raw to raw
# qemu-img measure -f raw -O raw test.raw 
required size: 4294967296
fully allocated size: 4294967296

# qemu-img convert -p -f raw -O raw test.raw /dev/vg/lv 
    (100.00/100%)
# qemu-img info /dev/vg/lv 
image: /dev/vg/lv
file format: raw
virtual size: 5.0G (5368709120 bytes)  ----------> size of lv is 5g, so it
disk size: 0                                       should be expected.

Comment 6 Stefan Hajnoczi 2017-10-11 16:07:34 UTC

(In reply to pingl from comment #5)
> According to the test result, file size calculated by "qemu-img measure" is
> greater than the actual value. Could you help to check whether it is
> expected?

Yes, it's fine is qemu-img measure reports larger sizes than the actual image.  The qemu-img documentation says: "The values reported are guaranteed to be large enough to fit the image".

Comment 7 Ping Li 2017-10-12 04:47:39 UTC

(In reply to Stefan Hajnoczi from comment #6)
> (In reply to pingl from comment #5)
> > According to the test result, file size calculated by "qemu-img measure" is
> > greater than the actual value. Could you help to check whether it is
> > expected?
> 
> Yes, it's fine is qemu-img measure reports larger sizes than the actual
> image.  The qemu-img documentation says: "The values reported are guaranteed
> to be large enough to fit the image".


I am still confused with the documentation in the qemu-img manpage[1].
For qcow2 format, It should be smaller?

[1]documentation
The "required size" is the file size of the new image.  It may be
smaller than the virtual disk size if the image format supports
compact representation.

Comment 8 Stefan Hajnoczi 2017-10-18 15:37:34 UTC

(In reply to pingl from comment #7)
> (In reply to Stefan Hajnoczi from comment #6)
> > (In reply to pingl from comment #5)
> > > According to the test result, file size calculated by "qemu-img measure" is
> > > greater than the actual value. Could you help to check whether it is
> > > expected?
> > 
> > Yes, it's fine is qemu-img measure reports larger sizes than the actual
> > image.  The qemu-img documentation says: "The values reported are guaranteed
> > to be large enough to fit the image".
> 
> 
> I am still confused with the documentation in the qemu-img manpage[1].
> For qcow2 format, It should be smaller?
> 
> [1]documentation
> The "required size" is the file size of the new image.  It may be
> smaller than the virtual disk size if the image format supports
> compact representation.

The comment about qcow2 refers to the fact that the qcow2 file for a 10 GB virtual disk might only require a host file of size 4 MB, for example, because qcow2 only stores clusters that have been allocated (it has a "compact representation").

That is different from saying that qemu-img measure may report a size that is larger than the actual size of the newly converted file.

Comment 10 Ping Li 2017-10-19 06:06:32 UTC

Closed the bug as verified according to the comment 5 and comment 9

Comment 14 errata-xmlrpc 2018-04-11 00:12:33 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1104

Note You need to log in before you can comment on or make changes to this bug.